# Lecture 8 Mathematics of Data: ISOMAP and LLE

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Lecture 8 Mathematics of Data: ISOMAP and LLE

2 John Dewey If knowledge comes from the impressions made upon us by natural objects, it is impossible to procure knowledge without the use of objects which impress the mind. Democracy and Educa.on: an introduc.on to the philosophy of educa.on, 1916

3 Matlab Dimensionality Reduction Toolbox h#p://homepage.tudel1.nl/19j49/ Matlab_Toolbox_for_Dimensionality_ReducDon.html Math.pku.edu.cn/teachers/yaoy/Spring2011/matlab/drtoolbox Principal Component Analysis (PCA), ProbabilisDc PC Factor Analysis (FA), Sammon mapping, Linear Discriminant Analysis (LDA) MulDdimensional scaling (MDS), Isomap, Landmark Isomap Local Linear Embedding (LLE), Laplacian Eigenmaps, Hessian LLE, Conformal Eigenmaps Local Tangent Space Alignment (LTSA), Maximum Variance Unfolding (extension of LLE) Landmark MVU (LandmarkMVU), Fast Maximum Variance Unfolding (FastMVU) Kernel PCA Diffusion maps

4 Recall: PCA Principal Component Analysis (PCA) X p n = [X 1 X 2... X n ] One Dimensional Manifold

5 Recall: MDS Given pairwise distances D, where D ij = d ij2, the squared distance between point i and j Convert the pairwise distance matrix D (c.n.d.) into the dot product matrix B (p.s.d.) B ij (a) = -.5 H(a) D H (a), Hölder matrix H(a) = I- 1a ; a = 1 k : B ij = -.5 (D ij - D ik D jk ) a = 1/n: N N B ij = 1 2 D ij 1 N D sj 1 N D it + 1 s=1 t =1 EigendecomposiDon of B = YY T N 2 N s,t =1 D st If we preserve the pairwise Euclidean distances do we preserve the structure??

6 Nonlinear Manifolds.. A PCA and MDS see the Euclidean distance What is important is the geodesic distance Unfold the manifold

7 Intrinsic Description.. To preserve the geodesic distance and not the Euclidean distance.

8 Two Basic Geometric Embedding Methods Tenenbaum-de Silva-Langford Isomap Algorithm Global approach. On a low dimensional embedding Nearby points should be nearby. Faraway points should be faraway. Roweis-Saul Locally Linear Embedding Algorithm Local approach Nearby points nearby

9 Isomap Estimate the geodesic distance between faraway points. For neighboring points Euclidean distance is a good approximation to the geodesic distance. For faraway points estimate the distance by a series of short hops between neighboring points. Find shortest paths in a graph with edges connecting neighboring data points Once we have all pairwise geodesic distances use classical metric MDS

10 Isomap - Algorithm Determine the neighbors. All points in a fixed radius. K nearest neighbors Construct a neighborhood graph. Each point is connected to the other if it is a K nearest neighbor. Edge Length equals the Euclidean distance Compute the shortest paths between two nodes Floyd s Algorithm (O(N 3 )) Dijkstra s Algorithm (O(kN 2 logn)) Construct a lower dimensional embedding. Classical MDS

11 Isomap

12 Example

13

14

15 Residual Variance Face Images SwisRoll Hand Images 2

16 ISOMAP on Alanine-dipeptide ISOMAP 3D embedding with RMSD metric on 3900 Kcenters

17 Theory of ISOMAP ISOMAP has provable convergence guarantees; Given that {x i } is sampled sufficiently dense, ISOMAP will approximate closely the original distance as measured in manifold M; In other words, actual geodesic distance approximadons using graph G can be arbitrarily good; Let s examine these theoredcal guarantees in more detail

18 Possible Issues

20

21 Dense- sampling Theorem [Bernstein, de Silva, Langford, and Tenenbaum 2000]

22 Proof Proof of Theorem 1 Proof: d M (x, y) d S (x, y) (1 + 4δ/)d M (x, y) The left hand side of the inequality follows directly from the triangle inequality. Let γ be any piecewise-smooth arc connecting x to y with = length(γ). If 2δ then x and y are connected by an edge in G which we can use as our path.

23 Proof

24 Proof

25 The Second ApproximaDon d S d G We would like to now show the other approximate equality: d S d G. First let s make some definitions: 1. The minimum radius of curvature r 0 = r 0 (M) is defined by 1 r 0 = max γ,t γ (t) where γ varies over all unit-speed geodesics in M and t is in the domain D of γ. Intuitively, geodesics in M curl around less tightly than circles of radius less than r 0 (M). 2. The minimum branch separation s 0 = s 0 (M) is the largest positive number for which x y < s 0 implies d M (x, y) πr 0 for any x, y M. Lemma: If γ is a geodesic in M connecting points x and y, and if = length(γ) πr 0, then: 2r 0 sin(/2r 0 ) x y

26 Remarks We will take this Lemma without proof as it is somewhat technical and long. Using the fact that sin(t) t t 3 /6 for t 0 we can write down a weakened form of the Lemma: (1 2 /24r0 2 ) x y We can also write down an even more weakened version valid for πr 0 : We can now show d G d S. (2/π) x y

27 Theorem 2 [Bernstein, de Silva, Langford, and Tenenbaum 2000] Theorem 2: Let λ > 0 be given. Suppose data points x i, x i+1 M satisfy: x i x i+1 < s 0 x i x i+1 (2/π)r 0 24λ Suppose also there is a geodesic arc of length = d M (x i, x i+1 ) connecting x i to x i+1. Then: (1 λ) x i x i+1

28 Proof Proof of Theorem 2 By the first assumption we can directly conclude πr 0. This fact allows us to apply the Lemma using the weakest form combined with the second assumption gives us: (π/2) x i x i+1 r 0 24λ Solving for λ in the above gives: 1 λ (1 2 /24r0 2 ). Applying the weakened statement of the Lemma then gives us the desired result. Combining Theorem 1 and 2 shows d M d G. This leads us then to our main theorem...

29 Main Theorem [Bernstein, de Silva, Langford, and Tenenbaum 2000] Theorem 1: Let M be a compact submanifold of R n and let {x i } be a finite set of data points in M. We are given a graph G on {x i } and positive real numbers λ 1, λ 2 < 1 and δ, > 0. Suppose: 1. G contains all edges (x i, x j ) of length x i x j. 2. The data set {x i } statisfies a δ-sampling condition for every point m M there exists an x i such that d M (m, x i ) < δ. 3. M is geodesically convex the shortest curve joining any two points on the surface is a geodesic curve. 4. < (2/π)r 0 24λ1, where r 0 is the minimum radius of curvature of M 1 r 0 = max γ,t γ (t) where γ varies over all unit-speed geodesics in M. 5. < s 0, where s 0 is the minimum branch separation of M the largest positive number for which x y < s 0 implies d M (x, y) πr δ < λ 2 /4. Then the following is valid for all x, y M, (1 λ 1 )d M (x, y) d G (x, y) (1 + λ 2 )d M (x, y)

30 ProbabilisDc Result So, short Euclidean distance hops along G approximate well actual geodesic distance as measured in M. What were the main assumptions we made? The biggest one was the δ-sampling density condition. A probabilistic version of the Main Theorem can be shown where each point x i is drawn from a density function. Then the approximation bounds will hold with high probability. Here s a truncated version of what the theorem looks like now: Asymptotic Convergence Theorem: Given λ 1, λ 2,µ>0 then for density function α sufficiently large: 1 λ 1 d G(x, y) d M (x, y) 1 + λ 2 will hold with probability at least 1 µ for any two data points x, y.

31 A Shortcoming of ISOMAP One need to compute pairwise hortest path etween all sample pairs (,j) Global on- sparse Cubic complexity O(N 3 )

32 Locally Linear Embedding manifold is a topological space which is locally Euclidean. Fit Locally, Think Globally

33 Fit Locally We expect each data point and its neighbours to lie on or close to a locally linear patch of the manifold. Each point can be written as a linear combination of its neighbors. The weights choosen to minimize the reconstruction Error. Derivation on board

34 Important property... The weights that minimize the reconstrucdon errors are invariant to rotadon, rescaling and transladon of the data points. Invariance to transladon is enforced by adding the constraint that the weights sum to one. The same weights that reconstruct the datapoints in D dimensions should reconstruct it in the manifold in d dimensions. The weights characterize the intrinsic geometric properdes of each neighborhood.

35 Think Globally

36 Algorithm (K-NN) Local fipng step (with centering): Consider a point x i Choose its K(i) neighbors η j whose origin is at x i Compute the (sum- to- one) weights w ij which minimizes Ψ i K(i) ( w) = x i w ij η j Contruct neighborhood inner product: C jk = η j,η k Compute the weight ector w i =(w ij ), where 1 is K- vector of ll- one and λ is a regularizadon parameter Then normalize w i to a sum- to- one vector. j =1 2, w ij =1, x i = 0 ( ) 1 1 w i = C + λi j

37 Algorithm (K-NN) Local fipng step (without centering): Consider a point x i Choose its K(i) neighbors x j Compute the (sum- to- one) weights w ij which minimizes Ψ i ( w) = x i w ij x j Contruct neighborhood inner product: C jk = η j,η k Compute the weight ector w i =(w ij ), where K(i) j =1 2, v ik = η k, x i w i = C + v i, v i = ( v ) ik R K(i)

38 Algorithm continued Global embedding step: Construct N- by- N weight matrix W: Compute d- by- N matrix Y which minimizes ( ) = Y i W ij Y j φ Y i N j =1 2 W ij = w ij, j N(i) 0, otherwise = Y(I W ) T (I W )Y T Compute: B = (I W ) T (I W ) Find d+1 bo#om eigenvectors of B, v n,v n- 1,,v n- d Let d- dimensional embedding Y =[ v n- 1,v n- 2, v n- 1 ]

39 Remarks on LLE Searching k- nearest neighbors is of O(kN) W is sparse, kn/n^2=k/n nozeros W might be negadve, addidonal nonnegadve constraint can be imposed B=(I- W) T (I- W) is posidve semi- definite (p.s.d.) Open Problem: exact reconstrucdon condidon?

40

41

42

43 Grolliers Encyclopedia

44 Summary.. ISOMAP Do MDS on the geodesic distance matrix. Global approach LLE Model local neighborhoods as linear a patches and then embed in a lower dimensional manifold. Local approach Might not work for nonconvex manifolds with holes Extensions: Landmark, Conformal & Isometric ISOMAP onconvex manifolds with holes xtensions: Hessian LLE Laplacian Eigenmaps etc Both needs manifold finely sampled

45 Landmark (Sparse) ISOMAP ISOMAP out of the box is not scalable. Two bottlenecks: All pairs shortest path - O(kN 2 log N). MDS eigenvalue calculation on a full NxN matrix - O(N 3 ). For contrast, LLE is limited by a sparse eigenvalue computation - O(dN 2 ). Landmark ISOMAP (L-ISOMAP) Idea: Use n << N landmark points from {x i } and compute a nxn matrix of geodesic distances, D n, from each data point to the landmark points only. Use new procedure Landmark-MDS (LMDS) to find a Euclidean embedding of all the data utilizes idea of triangulation similar to GPS. Savings: L-ISOMAP will have shortest paths calculation of O(knN log N) and LMDS eigenvalue problem of O(n 2 N).

46 Landmark MDS (Restriction) 1. Designate a set of n landmark points. 2. Apply classical MDS to the nxnmatrix n of the squared distances between each landmark point to find a d-dimensional embedding of these n points. Let L k be the dxnmatrix containing the embedded landmark points constructed by utilizing the calculated eigenvectors v i and eigenvalues λ i. λ1 v T 1 λ2 v T 2 L k =. λ d v T d

47 LMDS (Extension) 3. Apply distance-based triangulation to find a d-dimensional embedding of all N points. Let δ 1,..., δ n be vectors of the squared distances from the i-th landmark to all the landmarks and let δ µ be the mean of these vectors. Let δ x be the vector of squared distances between a point x and the landmark points. Then the i-th component of the embedding vector for y x is: y i x = 1 2 v T i λi ( δ x δ µ ) It can be shown that the above embedding of y x is equivalent to projecting onto the first d principal components of the landmarks. 4. Finally, we can optionally choose to run PCA to reorient our axes.

48 Landmark Choice How many landmark points should we choose?... d + 1 landmarks are enough for the triangulation to locate each point uniquely, but heuristics show that a few more is better for stability. Poorly distributed landmarks could lead to foreshortening projection onto the d-dimensional subspace causes a shortening of distances. Good methods are random OR use more expensive MinMax method that for each new landmark added maximizes the minimum distance to the already chosen ones. Either way, running L-ISOMAP in combination with cross-validation techniques would be useful to find a stable embedding.

49 Further exploration yet Hierarchical landmarks: cover- tree Nyström method

50 L-ISOMAP Examples

51 Generative Models in Manifold Learning

52 Conformal & Isometric Embedding

53 Isometric and Conformal Isometric mapping Intrinsically flat manifold Invariants Geodesic distances are reserved. Metric space under geodesic distance. Conformal Embedding Locally isometric upto a scale factor s(y) EsDmate s(y) and rescale. C- Isomap Original data should be uniformly dense

54

55 Linear, Isometric, Conformal If f is a linear isometry f : R d R D then we can simply use PCA or MDS to recover the d significant dimensions Plane. If f is an isometric embedding f : Y R D then provided that data points are sufficiently dense and Y R d is a convex domain we can use ISOMAP to recover the approximate original structure Swiss Roll. If f is a conformal embedding f : Y R D then we must assume the data is uniformly dense in Y and Y R d is a convex domain and then we can successfuly use C-ISOMAP Fish Bowl.

56 Conformal Isomap Idea behind C-ISOMAP: Not only estimate geodesic distances, but also scalar function s(y). Let µ(i) be the mean distance from xi to its k-nn. Each yi and its k-nn occupy a d-dimensional disk of radius r r depends only on d and sampling density. f maps this disk to approximately a d-dimensional disk on M of radius s(y i )r µ(i) s(y i ). µ(i) is a reasonable estimate of s(yi ) since it will be off by a constant factor (uniform density assumption).

57 C-Isomap We replace each edge weight in G by x i x j / µ(i)µ(j). Everything else is the same. Resulting Effect: magnify regions of high density and shrink regions of low density. A similar convergence theorem as given before can be shown about C-ISOMAP assuming that Y is sampled uniformly from a bounded convex region.

58 C-Isomap Example I We will compare LLE, ISOMAP, C-ISOMAP, and MDS on toy datasets. Conformal Fishbowl: Use stereographic projection to project points uniformly distributed in a disk in R 2 onto a sphere with the top removed. Uniform Fishbowl: Points distributed uniformly on the surface of the fishbowl. Offset Fishbowl: Same as conformal fishbowl but points are sampled in Y with a Gaussian offset from center.

59 C-Isomap Example I

60 C-Isomap Example II 2000 face images were randomly generated varying in distance and left-right pose. Each image is a vector in dimensional space. Below shows the four extreme cases. Conformal because changes in orientation at a long distance will have a smaller effect on local pixel distances than the corresponding change at a shorter distance.

61 C-Isomap Example II C-ISOMAP separates the two intrinsic dimensions cleanly. ISOMAP narrows as faces get further away. LLE is highly distorted.

62 Remark

63 Recap and Problems LLE ISOMAP Approach Local Global Isometry Most of the time, covariance distortion Yes Conformal No Guarantees, but sometimes C-ISOMAP Speed Quadratic in N Cubic in N, but L-ISOMAP How do LLE and L-ISOMAP compare in the quality of their output on real world datasets? can we develop a quantitative metric to evaluate them? How much improvement in classification tasks do NLDR techniques really give over traditional dimensionality reduction techniques? Is there some sort of heuristic for choosing k? Possibly could we utilize heirarchical clustering information in constructing a better graph G? Lots of research potential...

64 Reference Tenenbaum, de Silva, and Langford, A Global Geometric Framework for Nonlinear Dimensionality ReducDon. Science 290: , 22 Dec Roweis and Saul, Nonlinear Dimensionality ReducDon by Locally Linear Embedding. Science 290: , 22 Dec M. Bernstein, V. de Silva, J. Langford, and J. Tenenbaum. Graph ApproximaDons to Geodesics on Embedded Manifolds. Technical Report, Department of Psychology, Stanford University, V. de Silva and J.B. Tenenbaum. Global versus local methods in nonlinear dimensionality reducdon. Neural InformaDon Processing Systems 15 (NIPS 2002), pp , V. de Silva and J.B. Tenenbaum. Unsupervised learning of curved manifolds. Nonlinear EsDmaDon and ClassificaDon, V. de Silva and J.B. Tenenbaum. Sparse mulddimensional scaling using landmark points. Available at: h#p://math.stanford.edu/~silva/public/publicadons.html

65 Slides stolen from Epnger, Vikas C. Raykar Vin de Silva.

### Global versus local methods in nonlinear dimensionality reduction

Global versus local methods in nonlinear dimensionality reduction Vin de Silva Department of Mathematics, Stanford University, Stanford. CA 94305 silva@math.stanford.edu Joshua B. Tenenbaum Department

### Global versus local methods in nonlinear dimensionality reduction

Global versus local methods in nonlinear dimensionality reduction Vin de Silva Department of Mathematics, Stanford University, Stanford. CA 94305 silva@math.stanford.edu Joshua B. Tenenbaum Department

### Dimension Reduction CS534

Dimension Reduction CS534 Why dimension reduction? High dimensionality large number of features E.g., documents represented by thousands of words, millions of bigrams Images represented by thousands of

### Data fusion and multi-cue data matching using diffusion maps

Data fusion and multi-cue data matching using diffusion maps Stéphane Lafon Collaborators: Raphy Coifman, Andreas Glaser, Yosi Keller, Steven Zucker (Yale University) Part of this work was supported by

### Robust Pose Estimation using the SwissRanger SR-3000 Camera

Robust Pose Estimation using the SwissRanger SR- Camera Sigurjón Árni Guðmundsson, Rasmus Larsen and Bjarne K. Ersbøll Technical University of Denmark, Informatics and Mathematical Modelling. Building,

### Iterative Non-linear Dimensionality Reduction by Manifold Sculpting

Iterative Non-linear Dimensionality Reduction by Manifold Sculpting Mike Gashler, Dan Ventura, and Tony Martinez Brigham Young University Provo, UT 84604 Abstract Many algorithms have been recently developed

### SELECTION OF THE OPTIMAL PARAMETER VALUE FOR THE LOCALLY LINEAR EMBEDDING ALGORITHM. Olga Kouropteva, Oleg Okun and Matti Pietikäinen

SELECTION OF THE OPTIMAL PARAMETER VALUE FOR THE LOCALLY LINEAR EMBEDDING ALGORITHM Olga Kouropteva, Oleg Okun and Matti Pietikäinen Machine Vision Group, Infotech Oulu and Department of Electrical and

### Cluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1

Cluster Analysis Mu-Chun Su Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Introduction Cluster analysis is the formal study of algorithms and methods

### Selecting Models from Videos for Appearance-Based Face Recognition

Selecting Models from Videos for Appearance-Based Face Recognition Abdenour Hadid and Matti Pietikäinen Machine Vision Group Infotech Oulu and Department of Electrical and Information Engineering P.O.

### Dimension Reduction of Image Manifolds

Dimension Reduction of Image Manifolds Arian Maleki Department of Electrical Engineering Stanford University Stanford, CA, 9435, USA E-mail: arianm@stanford.edu I. INTRODUCTION Dimension reduction of datasets

### Correspondence. CS 468 Geometry Processing Algorithms. Maks Ovsjanikov

Shape Matching & Correspondence CS 468 Geometry Processing Algorithms Maks Ovsjanikov Wednesday, October 27 th 2010 Overall Goal Given two shapes, find correspondences between them. Overall Goal Given

### CIE L*a*b* color model

CIE L*a*b* color model To further strengthen the correlation between the color model and human perception, we apply the following non-linear transformation: with where (X n,y n,z n ) are the tristimulus

### RDRToolbox A package for nonlinear dimension reduction with Isomap and LLE.

RDRToolbox A package for nonlinear dimension reduction with Isomap and LLE. Christoph Bartenhagen October 30, 2017 Contents 1 Introduction 1 1.1 Loading the package......................................

### Topological estimation using witness complexes. Vin de Silva, Stanford University

Topological estimation using witness complexes, Acknowledgements Gunnar Carlsson (Mathematics, Stanford) principal collaborator Afra Zomorodian (CS/Robotics, Stanford) persistent homology software Josh

### Clustering Data That Resides on a Low-Dimensional Manifold in a High-Dimensional Measurement Space. Avinash Kak Purdue University

Clustering Data That Resides on a Low-Dimensional Manifold in a High-Dimensional Measurement Space Avinash Kak Purdue University February 15, 2016 10:37am An RVL Tutorial Presentation Originally presented

### CSE 6242 A / CS 4803 DVA. Feb 12, Dimension Reduction. Guest Lecturer: Jaegul Choo

CSE 6242 A / CS 4803 DVA Feb 12, 2013 Dimension Reduction Guest Lecturer: Jaegul Choo CSE 6242 A / CS 4803 DVA Feb 12, 2013 Dimension Reduction Guest Lecturer: Jaegul Choo Data is Too Big To Do Something..

### Dimension reduction for hyperspectral imaging using laplacian eigenmaps and randomized principal component analysis:midyear Report

Dimension reduction for hyperspectral imaging using laplacian eigenmaps and randomized principal component analysis:midyear Report Yiran Li yl534@math.umd.edu Advisor: Wojtek Czaja wojtek@math.umd.edu

### Definition A metric space is proper if all closed balls are compact. The length pseudo metric of a metric space X is given by.

Chapter 1 Geometry: Nuts and Bolts 1.1 Metric Spaces Definition 1.1.1. A metric space is proper if all closed balls are compact. The length pseudo metric of a metric space X is given by (x, y) inf p. p:x

### Image Processing. Image Features

Image Processing Image Features Preliminaries 2 What are Image Features? Anything. What they are used for? Some statements about image fragments (patches) recognition Search for similar patches matching

### Data Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University

Data Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University Exploratory data analysis tasks Examine the data, in search of structures

### Differential Structure in non-linear Image Embedding Functions

Differential Structure in non-linear Image Embedding Functions Robert Pless Department of Computer Science, Washington University in St. Louis pless@cse.wustl.edu Abstract Many natural image sets are samples

### LOCAL LINEAR APPROXIMATION OF PRINCIPAL CURVE PROJECTIONS. Peng Zhang, Esra Ataer-Cansizoglu, Deniz Erdogmus

2012 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 23 26, 2012, SATANDER, SPAIN LOCAL LINEAR APPROXIMATION OF PRINCIPAL CURVE PROJECTIONS Peng Zhang, Esra Ataer-Cansizoglu,

### Detection and approximation of linear structures in metric spaces

Stanford - July 12, 2012 MMDS 2012 Detection and approximation of linear structures in metric spaces Frédéric Chazal Geometrica group INRIA Saclay Joint work with M. Aanjaneya, D. Chen, M. Glisse, L. Guibas,

### 274 Curves on Surfaces, Lecture 5

274 Curves on Surfaces, Lecture 5 Dylan Thurston Notes by Qiaochu Yuan Fall 2012 5 Ideal polygons Previously we discussed three models of the hyperbolic plane: the Poincaré disk, the upper half-plane,

### Greedy Routing with Guaranteed Delivery Using Ricci Flow

Greedy Routing with Guaranteed Delivery Using Ricci Flow Jie Gao Stony Brook University Joint work with Rik Sarkar, Xiaotian Yin, Wei Zeng, Feng Luo, Xianfeng David Gu Greedy Routing Assign coordinatesto

### Local Linear Embedding. Katelyn Stringer ASTR 689 December 1, 2015

Local Linear Embedding Katelyn Stringer ASTR 689 December 1, 2015 Idea Behind LLE Good at making nonlinear high-dimensional data easier for computers to analyze Example: A high-dimensional surface Think

### Edge and local feature detection - 2. Importance of edge detection in computer vision

Edge and local feature detection Gradient based edge detection Edge detection by function fitting Second derivative edge detectors Edge linking and the construction of the chain graph Edge and local feature

### Estimating Cluster Overlap on Manifolds and its Application to Neuropsychiatric Disorders

Estimating Cluster Overlap on Manifolds and its Application to Neuropsychiatric Disorders Peng Wang 1, Christian Kohler 2, Ragini Verma 1 1 Department of Radiology University of Pennsylvania Philadelphia,

### LEARNING ON STATISTICAL MANIFOLDS FOR CLUSTERING AND VISUALIZATION

LEARNING ON STATISTICAL MANIFOLDS FOR CLUSTERING AND VISUALIZATION Kevin M. Carter 1, Raviv Raich, and Alfred O. Hero III 1 1 Department of EECS, University of Michigan, Ann Arbor, MI 4819 School of EECS,

### CAT(0) BOUNDARIES OF TRUNCATED HYPERBOLIC SPACE

CAT(0) BOUNDARIES OF TRUNCATED HYPERBOLIC SPACE KIM RUANE Abstract. We prove that the CAT(0) boundary of a truncated hyperbolic space is homeomorphic to a sphere with disks removed. In dimension three,

### Texture Mapping using Surface Flattening via Multi-Dimensional Scaling

Texture Mapping using Surface Flattening via Multi-Dimensional Scaling Gil Zigelman Ron Kimmel Department of Computer Science, Technion, Haifa 32000, Israel and Nahum Kiryati Department of Electrical Engineering

### Head Frontal-View Identification Using Extended LLE

Head Frontal-View Identification Using Extended LLE Chao Wang Center for Spoken Language Understanding, Oregon Health and Science University Abstract Automatic head frontal-view identification is challenging

### Unsupervised Learning

Networks for Pattern Recognition, 2014 Networks for Single Linkage K-Means Soft DBSCAN PCA Networks for Kohonen Maps Linear Vector Quantization Networks for Problems/Approaches in Machine Learning Supervised

### SOLVING PARTIAL DIFFERENTIAL EQUATIONS ON POINT CLOUDS

SOLVING PARTIAL DIFFERENTIAL EQUATIONS ON POINT CLOUDS JIAN LIANG AND HONGKAI ZHAO Abstract. In this paper we present a general framework for solving partial differential equations on manifolds represented

### The Pre-Image Problem in Kernel Methods

The Pre-Image Problem in Kernel Methods James Kwok Ivor Tsang Department of Computer Science Hong Kong University of Science and Technology Hong Kong The Pre-Image Problem in Kernel Methods ICML-2003 1

### Non-linear CCA and PCA by Alignment of Local Models

Non-linear CCA and PCA by Alignment of Local Models Jakob J. Verbeek, Sam T. Roweis, and Nikos Vlassis Informatics Institute, University of Amsterdam Department of Computer Science,University of Toronto

### The Farey Tessellation

The Farey Tessellation Seminar: Geometric Structures on manifolds Mareike Pfeil supervised by Dr. Gye-Seon Lee 15.12.2015 Introduction In this paper, we are going to introduce the Farey tessellation. Since

### Réduction de modèles pour des problèmes d optimisation et d identification en calcul de structures

Réduction de modèles pour des problèmes d optimisation et d identification en calcul de structures Piotr Breitkopf Rajan Filomeno Coelho, Huanhuan Gao, Anna Madra, Liang Meng, Guénhaël Le Quilliec, Balaji

### LARGE dimensionality is a significant problem for many

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS PART B: CYBERNETICS 1 Manifold Learning by Graduated Optimization Michael Gashler, Dan Ventura, and Tony Martinez Abstract We present an algorithm for

### Non-Local Estimation of Manifold Structure

Non-Local Estimation of Manifold Structure Yoshua Bengio, Martin Monperrus and Hugo Larochelle Département d Informatique et Recherche Opérationnelle Centre de Recherches Mathématiques Université de Montréal

### Data Preprocessing. Javier Béjar AMLT /2017 CS - MAI. (CS - MAI) Data Preprocessing AMLT / / 71 BY: \$\

Data Preprocessing S - MAI AMLT - 2016/2017 (S - MAI) Data Preprocessing AMLT - 2016/2017 1 / 71 Outline 1 Introduction Data Representation 2 Data Preprocessing Outliers Missing Values Normalization Discretization

### CPSC 340: Machine Learning and Data Mining. Kernel Trick Fall 2017

CPSC 340: Machine Learning and Data Mining Kernel Trick Fall 2017 Admin Assignment 3: Due Friday. Midterm: Can view your exam during instructor office hours or after class this week. Digression: the other

### Extended Isomap for Pattern Classification

From: AAAI- Proceedings. Copyright, AAAI (www.aaai.org). All rights reserved. Extended for Pattern Classification Ming-Hsuan Yang Honda Fundamental Research Labs Mountain View, CA 944 myang@hra.com Abstract

### Generalized Principal Component Analysis CVPR 2007

Generalized Principal Component Analysis Tutorial @ CVPR 2007 Yi Ma ECE Department University of Illinois Urbana Champaign René Vidal Center for Imaging Science Institute for Computational Medicine Johns

### Computer vision: models, learning and inference. Chapter 13 Image preprocessing and feature extraction

Computer vision: models, learning and inference Chapter 13 Image preprocessing and feature extraction Preprocessing The goal of pre-processing is to try to reduce unwanted variation in image due to lighting,

### Towards Multi-scale Heat Kernel Signatures for Point Cloud Models of Engineering Artifacts

Towards Multi-scale Heat Kernel Signatures for Point Cloud Models of Engineering Artifacts Reed M. Williams and Horea T. Ilieş Department of Mechanical Engineering University of Connecticut Storrs, CT

### A Stochastic Optimization Approach for Unsupervised Kernel Regression

A Stochastic Optimization Approach for Unsupervised Kernel Regression Oliver Kramer Institute of Structural Mechanics Bauhaus-University Weimar oliver.kramer@uni-weimar.de Fabian Gieseke Institute of Structural

### Surface Reconstruction with MLS

Surface Reconstruction with MLS Tobias Martin CS7960, Spring 2006, Feb 23 Literature An Adaptive MLS Surface for Reconstruction with Guarantees, T. K. Dey and J. Sun A Sampling Theorem for MLS Surfaces,

### Subdivision Surfaces

Subdivision Surfaces 1 Geometric Modeling Sometimes need more than polygon meshes Smooth surfaces Traditional geometric modeling used NURBS Non uniform rational B-Spline Demo 2 Problems with NURBS A single

### Indian Institute of Technology Kanpur. Visuomotor Learning Using Image Manifolds: ST-GK Problem

Indian Institute of Technology Kanpur Introduction to Cognitive Science SE367A Visuomotor Learning Using Image Manifolds: ST-GK Problem Author: Anurag Misra Department of Computer Science and Engineering

### A Taxonomy of Semi-Supervised Learning Algorithms

A Taxonomy of Semi-Supervised Learning Algorithms Olivier Chapelle Max Planck Institute for Biological Cybernetics December 2005 Outline 1 Introduction 2 Generative models 3 Low density separation 4 Graph

### Hyperbolic structures and triangulations

CHAPTER Hyperbolic structures and triangulations In chapter 3, we learned that hyperbolic structures lead to developing maps and holonomy, and that the developing map is a covering map if and only if the

### Inverse and Implicit functions

CHAPTER 3 Inverse and Implicit functions. Inverse Functions and Coordinate Changes Let U R d be a domain. Theorem. (Inverse function theorem). If ϕ : U R d is differentiable at a and Dϕ a is invertible,

### Non-Local Estimation of Manifold Structure

Neural Computation Manuscript #3171 Non-Local Estimation of Manifold Structure Yoshua Bengio, Martin Monperrus and Hugo Larochelle Département d Informatique et Recherche Opérationnelle Centre de Recherches

### Shape spaces. Shape usually defined explicitly to be residual: independent of size.

Shape spaces Before define a shape space, must define shape. Numerous definitions of shape in relation to size. Many definitions of size (which we will explicitly define later). Shape usually defined explicitly

### Graphs associated to CAT(0) cube complexes

Graphs associated to CAT(0) cube complexes Mark Hagen McGill University Cornell Topology Seminar, 15 November 2011 Outline Background on CAT(0) cube complexes The contact graph: a combinatorial invariant

### Semi-supervised Learning with Density Based Distances

Semi-supervised Learning with Density Based Distances Avleen S. Bijral Toyota Technological Institute Chicago, IL 6637 Nathan Ratliff Google Inc. Pittsburg, PA 526 Nathan Srebro Toyota Technological Institute

### THE recovery of low-dimensional structures in data sets. Out-of-sample generalizations for supervised manifold learning for classification

1 Out-of-sample generalizations for supervised manifold learning for classification Elif Vural and Christine Guillemot arxiv:152.241v1 [cs.cv] 9 Feb 215 Abstract Supervised manifold learning methods for

### Classification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University

Classification Vladimir Curic Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University Outline An overview on classification Basics of classification How to choose appropriate

### The exam is closed book, closed notes except your one-page (two-sided) cheat sheet.

CS 189 Spring 2015 Introduction to Machine Learning Final You have 2 hours 50 minutes for the exam. The exam is closed book, closed notes except your one-page (two-sided) cheat sheet. No calculators or

### Surface Parameterization

Surface Parameterization A Tutorial and Survey Michael Floater and Kai Hormann Presented by Afra Zomorodian CS 468 10/19/5 1 Problem 1-1 mapping from domain to surface Original application: Texture mapping

### Application of Spectral Clustering Algorithm

1/27 Application of Spectral Clustering Algorithm Danielle Middlebrooks dmiddle1@math.umd.edu Advisor: Kasso Okoudjou kasso@umd.edu Department of Mathematics University of Maryland- College Park Advance

### Spectral Clustering. Presented by Eldad Rubinstein Based on a Tutorial by Ulrike von Luxburg TAU Big Data Processing Seminar December 14, 2014

Spectral Clustering Presented by Eldad Rubinstein Based on a Tutorial by Ulrike von Luxburg TAU Big Data Processing Seminar December 14, 2014 What are we going to talk about? Introduction Clustering and

### Two Connections between Combinatorial and Differential Geometry

Two Connections between Combinatorial and Differential Geometry John M. Sullivan Institut für Mathematik, Technische Universität Berlin Berlin Mathematical School DFG Research Group Polyhedral Surfaces

### An Efficient Algorithm for Local Distance Metric Learning

An Efficient Algorithm for Local Distance Metric Learning Liu Yang and Rong Jin Michigan State University Dept. of Computer Science & Engineering East Lansing, MI 48824 {yangliu1, rongjin}@cse.msu.edu

### Clustering. SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic

Clustering SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic Clustering is one of the fundamental and ubiquitous tasks in exploratory data analysis a first intuition about the

### A GENTLE INTRODUCTION TO THE BASIC CONCEPTS OF SHAPE SPACE AND SHAPE STATISTICS

A GENTLE INTRODUCTION TO THE BASIC CONCEPTS OF SHAPE SPACE AND SHAPE STATISTICS HEMANT D. TAGARE. Introduction. Shape is a prominent visual feature in many images. Unfortunately, the mathematical theory

### WITH the wide usage of information technology in almost. Supervised Nonlinear Dimensionality Reduction for Visualization and Classification

1098 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS PART B: CYBERNETICS, VOL. 35, NO. 6, DECEMBER 2005 Supervised Nonlinear Dimensionality Reduction for Visualization and Classification Xin Geng, De-Chuan

### Hyperspectral image segmentation using spatial-spectral graphs

Hyperspectral image segmentation using spatial-spectral graphs David B. Gillis* and Jeffrey H. Bowles Naval Research Laboratory, Remote Sensing Division, Washington, DC 20375 ABSTRACT Spectral graph theory

### Mapping a manifold of perceptual observations

Mapping a manifold of perceptual observations Joshua. Tenenbaum Department of rain and Cognitive Sciences Massachusetts Institute of Technology, Cambridge, M 2139 jbt@psyche.mit.edu bstract Nonlinear dimensionality

### Semi-supervised Regression using Hessian Energy with an Application to Semi-supervised Dimensionality Reduction

Semi-supervised Regression using Hessian Energy with an Application to Semi-supervised Dimensionality Reduction Kwang In Kim, Florian Steinke,3, and Matthias Hein Department of Computer Science, Saarland

### Nonlinear dimensionality reduction of large datasets for data exploration

Data Mining VII: Data, Text and Web Mining and their Business Applications 3 Nonlinear dimensionality reduction of large datasets for data exploration V. Tomenko & V. Popov Wessex Institute of Technology,

### Clustering and Visualisation of Data

Clustering and Visualisation of Data Hiroshi Shimodaira January-March 28 Cluster analysis aims to partition a data set into meaningful or useful groups, based on distances between data points. In some

### Lecture 2 The k-means clustering problem

CSE 29: Unsupervised learning Spring 2008 Lecture 2 The -means clustering problem 2. The -means cost function Last time we saw the -center problem, in which the input is a set S of data points and the

### Topic: Orientation, Surfaces, and Euler characteristic

Topic: Orientation, Surfaces, and Euler characteristic The material in these notes is motivated by Chapter 2 of Cromwell. A source I used for smooth manifolds is do Carmo s Riemannian Geometry. Ideas of

### Surface Reconstruction from Unorganized Points

Survey of Methods in Computer Graphics: Surface Reconstruction from Unorganized Points H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, W. Stuetzle SIGGRAPH 1992. Article and Additional Material at: http://research.microsoft.com/en-us/um/people/hoppe/proj/recon/

### Introduction to geometry

1 2 Manifolds A topological space in which every point has a neighborhood homeomorphic to (topological disc) is called an n-dimensional (or n-) manifold Introduction to geometry The German way 2-manifold

### Planes Intersecting Cones: Static Hypertext Version

Page 1 of 12 Planes Intersecting Cones: Static Hypertext Version On this page, we develop some of the details of the plane-slicing-cone picture discussed in the introduction. The relationship between the

### Research in Computational Differential Geomet

Research in Computational Differential Geometry November 5, 2014 Approximations Often we have a series of approximations which we think are getting close to looking like some shape. Approximations Often

### PATTERN CLASSIFICATION AND SCENE ANALYSIS

PATTERN CLASSIFICATION AND SCENE ANALYSIS RICHARD O. DUDA PETER E. HART Stanford Research Institute, Menlo Park, California A WILEY-INTERSCIENCE PUBLICATION JOHN WILEY & SONS New York Chichester Brisbane

### arxiv:math/ v3 [math.dg] 23 Jul 2007

THE INTRINSIC GEOMETRY OF A JORDAN DOMAIN arxiv:math/0512622v3 [math.dg] 23 Jul 2007 RICHARD L. BISHOP Abstract. For a Jordan domain in the plane the length metric space of points connected to an interior

### 5. THE ISOPERIMETRIC PROBLEM

Math 501 - Differential Geometry Herman Gluck March 1, 2012 5. THE ISOPERIMETRIC PROBLEM Theorem. Let C be a simple closed curve in the plane with length L and bounding a region of area A. Then L 2 4 A,

### A non-negative representation learning algorithm for selecting neighbors

Mach Learn (2016) 102:133 153 DOI 10.1007/s10994-015-5501-4 A non-negative representation learning algorithm for selecting neighbors Lili Li 1 Jiancheng Lv 1 Zhang Yi 1 Received: 8 August 2014 / Accepted:

### Supervised vs unsupervised clustering

Classification Supervised vs unsupervised clustering Cluster analysis: Classes are not known a- priori. Classification: Classes are defined a-priori Sometimes called supervised clustering Extract useful

### Lecture 2. Topology of Sets in R n. August 27, 2008

Lecture 2 Topology of Sets in R n August 27, 2008 Outline Vectors, Matrices, Norms, Convergence Open and Closed Sets Special Sets: Subspace, Affine Set, Cone, Convex Set Special Convex Sets: Hyperplane,

### Shape Classification and Cell Movement in 3D Matrix Tutorial (Part I)

Shape Classification and Cell Movement in 3D Matrix Tutorial (Part I) Fred Park UCI icamp 2011 Outline 1. Motivation and Shape Definition 2. Shape Descriptors 3. Classification 4. Applications: Shape Matching,

### Outline. Advanced Digital Image Processing and Others. Importance of Segmentation (Cont.) Importance of Segmentation

Advanced Digital Image Processing and Others Xiaojun Qi -- REU Site Program in CVIP (7 Summer) Outline Segmentation Strategies and Data Structures Algorithms Overview K-Means Algorithm Hidden Markov Model

### Algebraic Geometry of Segmentation and Tracking

Ma191b Winter 2017 Geometry of Neuroscience Geometry of lines in 3-space and Segmentation and Tracking This lecture is based on the papers: Reference: Marco Pellegrini, Ray shooting and lines in space.

### MultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A

MultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A. 205-206 Pietro Guccione, PhD DEI - DIPARTIMENTO DI INGEGNERIA ELETTRICA E DELL INFORMAZIONE POLITECNICO DI BARI

### Advanced Topics In Machine Learning Project Report : Low Dimensional Embedding of a Pose Collection Fabian Prada

Advanced Topics In Machine Learning Project Report : Low Dimensional Embedding of a Pose Collection Fabian Prada 1 Introduction In this project we present an overview of (1) low dimensional embedding,

### Manifold Alignment. Chang Wang, Peter Krafft, and Sridhar Mahadevan

Manifold Alignment Chang Wang, Peter Krafft, and Sridhar Mahadevan April 1, 211 2 Contents 5 Manifold Alignment 5 5.1 Introduction.................................... 5 5.1.1 Problem Statement............................

### An Introduction to Belyi Surfaces

An Introduction to Belyi Surfaces Matthew Stevenson December 16, 2013 We outline the basic theory of Belyi surfaces, up to Belyi s theorem (1979, [1]), which characterizes these spaces as precisely those

### Machine Learning: Think Big and Parallel

Day 1 Inderjit S. Dhillon Dept of Computer Science UT Austin CS395T: Topics in Multicore Programming Oct 1, 2013 Outline Scikit-learn: Machine Learning in Python Supervised Learning day1 Regression: Least

### Machine Learning : Clustering, Self-Organizing Maps

Machine Learning Clustering, Self-Organizing Maps 12/12/2013 Machine Learning : Clustering, Self-Organizing Maps Clustering The task: partition a set of objects into meaningful subsets (clusters). The

### LTI Thesis Defense: Riemannian Geometry and Statistical Machine Learning

Outline LTI Thesis Defense: Riemannian Geometry and Statistical Machine Learning Guy Lebanon Motivation Introduction Previous Work Committee:John Lafferty Geoff Gordon, Michael I. Jordan, Larry Wasserman

### 3 Nonlinear Regression

CSC 4 / CSC D / CSC C 3 Sometimes linear models are not sufficient to capture the real-world phenomena, and thus nonlinear models are necessary. In regression, all such models will have the same basic

### Diffusion Maps and Topological Data Analysis

Diffusion Maps and Topological Data Analysis Melissa R. McGuirl McGuirl (Brown University) Diffusion Maps and Topological Data Analysis 1 / 19 Introduction OVERVIEW Topological Data Analysis The use of

### Mathematical Programming and Research Methods (Part II)

Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types