Lecture 8 Mathematics of Data: ISOMAP and LLE
|
|
- Jane Carter
- 1 years ago
- Views:
Transcription
1 Lecture 8 Mathematics of Data: ISOMAP and LLE
2 John Dewey If knowledge comes from the impressions made upon us by natural objects, it is impossible to procure knowledge without the use of objects which impress the mind. Democracy and Educa.on: an introduc.on to the philosophy of educa.on, 1916
3 Matlab Dimensionality Reduction Toolbox h#p://homepage.tudel1.nl/19j49/ Matlab_Toolbox_for_Dimensionality_ReducDon.html Math.pku.edu.cn/teachers/yaoy/Spring2011/matlab/drtoolbox Principal Component Analysis (PCA), ProbabilisDc PC Factor Analysis (FA), Sammon mapping, Linear Discriminant Analysis (LDA) MulDdimensional scaling (MDS), Isomap, Landmark Isomap Local Linear Embedding (LLE), Laplacian Eigenmaps, Hessian LLE, Conformal Eigenmaps Local Tangent Space Alignment (LTSA), Maximum Variance Unfolding (extension of LLE) Landmark MVU (LandmarkMVU), Fast Maximum Variance Unfolding (FastMVU) Kernel PCA Diffusion maps
4 Recall: PCA Principal Component Analysis (PCA) X p n = [X 1 X 2... X n ] One Dimensional Manifold
5 Recall: MDS Given pairwise distances D, where D ij = d ij2, the squared distance between point i and j Convert the pairwise distance matrix D (c.n.d.) into the dot product matrix B (p.s.d.) B ij (a) = -.5 H(a) D H (a), Hölder matrix H(a) = I- 1a ; a = 1 k : B ij = -.5 (D ij - D ik D jk ) a = 1/n: N N B ij = 1 2 D ij 1 N D sj 1 N D it + 1 s=1 t =1 EigendecomposiDon of B = YY T N 2 N s,t =1 D st If we preserve the pairwise Euclidean distances do we preserve the structure??
6 Nonlinear Manifolds.. A PCA and MDS see the Euclidean distance What is important is the geodesic distance Unfold the manifold
7 Intrinsic Description.. To preserve the geodesic distance and not the Euclidean distance.
8 Two Basic Geometric Embedding Methods Tenenbaum-de Silva-Langford Isomap Algorithm Global approach. On a low dimensional embedding Nearby points should be nearby. Faraway points should be faraway. Roweis-Saul Locally Linear Embedding Algorithm Local approach Nearby points nearby
9 Isomap Estimate the geodesic distance between faraway points. For neighboring points Euclidean distance is a good approximation to the geodesic distance. For faraway points estimate the distance by a series of short hops between neighboring points. Find shortest paths in a graph with edges connecting neighboring data points Once we have all pairwise geodesic distances use classical metric MDS
10 Isomap - Algorithm Determine the neighbors. All points in a fixed radius. K nearest neighbors Construct a neighborhood graph. Each point is connected to the other if it is a K nearest neighbor. Edge Length equals the Euclidean distance Compute the shortest paths between two nodes Floyd s Algorithm (O(N 3 )) Dijkstra s Algorithm (O(kN 2 logn)) Construct a lower dimensional embedding. Classical MDS
11 Isomap
12 Example
13
14
15 Residual Variance Face Images SwisRoll Hand Images 2
16 ISOMAP on Alanine-dipeptide ISOMAP 3D embedding with RMSD metric on 3900 Kcenters
17 Theory of ISOMAP ISOMAP has provable convergence guarantees; Given that {x i } is sampled sufficiently dense, ISOMAP will approximate closely the original distance as measured in manifold M; In other words, actual geodesic distance approximadons using graph G can be arbitrarily good; Let s examine these theoredcal guarantees in more detail
18 Possible Issues
19 Two step approximadons
20
21 Dense- sampling Theorem [Bernstein, de Silva, Langford, and Tenenbaum 2000]
22 Proof Proof of Theorem 1 Proof: d M (x, y) d S (x, y) (1 + 4δ/)d M (x, y) The left hand side of the inequality follows directly from the triangle inequality. Let γ be any piecewise-smooth arc connecting x to y with = length(γ). If 2δ then x and y are connected by an edge in G which we can use as our path.
23 Proof
24 Proof
25 The Second ApproximaDon d S d G We would like to now show the other approximate equality: d S d G. First let s make some definitions: 1. The minimum radius of curvature r 0 = r 0 (M) is defined by 1 r 0 = max γ,t γ (t) where γ varies over all unit-speed geodesics in M and t is in the domain D of γ. Intuitively, geodesics in M curl around less tightly than circles of radius less than r 0 (M). 2. The minimum branch separation s 0 = s 0 (M) is the largest positive number for which x y < s 0 implies d M (x, y) πr 0 for any x, y M. Lemma: If γ is a geodesic in M connecting points x and y, and if = length(γ) πr 0, then: 2r 0 sin(/2r 0 ) x y
26 Remarks We will take this Lemma without proof as it is somewhat technical and long. Using the fact that sin(t) t t 3 /6 for t 0 we can write down a weakened form of the Lemma: (1 2 /24r0 2 ) x y We can also write down an even more weakened version valid for πr 0 : We can now show d G d S. (2/π) x y
27 Theorem 2 [Bernstein, de Silva, Langford, and Tenenbaum 2000] Theorem 2: Let λ > 0 be given. Suppose data points x i, x i+1 M satisfy: x i x i+1 < s 0 x i x i+1 (2/π)r 0 24λ Suppose also there is a geodesic arc of length = d M (x i, x i+1 ) connecting x i to x i+1. Then: (1 λ) x i x i+1
28 Proof Proof of Theorem 2 By the first assumption we can directly conclude πr 0. This fact allows us to apply the Lemma using the weakest form combined with the second assumption gives us: (π/2) x i x i+1 r 0 24λ Solving for λ in the above gives: 1 λ (1 2 /24r0 2 ). Applying the weakened statement of the Lemma then gives us the desired result. Combining Theorem 1 and 2 shows d M d G. This leads us then to our main theorem...
29 Main Theorem [Bernstein, de Silva, Langford, and Tenenbaum 2000] Theorem 1: Let M be a compact submanifold of R n and let {x i } be a finite set of data points in M. We are given a graph G on {x i } and positive real numbers λ 1, λ 2 < 1 and δ, > 0. Suppose: 1. G contains all edges (x i, x j ) of length x i x j. 2. The data set {x i } statisfies a δ-sampling condition for every point m M there exists an x i such that d M (m, x i ) < δ. 3. M is geodesically convex the shortest curve joining any two points on the surface is a geodesic curve. 4. < (2/π)r 0 24λ1, where r 0 is the minimum radius of curvature of M 1 r 0 = max γ,t γ (t) where γ varies over all unit-speed geodesics in M. 5. < s 0, where s 0 is the minimum branch separation of M the largest positive number for which x y < s 0 implies d M (x, y) πr δ < λ 2 /4. Then the following is valid for all x, y M, (1 λ 1 )d M (x, y) d G (x, y) (1 + λ 2 )d M (x, y)
30 ProbabilisDc Result So, short Euclidean distance hops along G approximate well actual geodesic distance as measured in M. What were the main assumptions we made? The biggest one was the δ-sampling density condition. A probabilistic version of the Main Theorem can be shown where each point x i is drawn from a density function. Then the approximation bounds will hold with high probability. Here s a truncated version of what the theorem looks like now: Asymptotic Convergence Theorem: Given λ 1, λ 2,µ>0 then for density function α sufficiently large: 1 λ 1 d G(x, y) d M (x, y) 1 + λ 2 will hold with probability at least 1 µ for any two data points x, y.
31 A Shortcoming of ISOMAP One need to compute pairwise hortest path etween all sample pairs (,j) Global on- sparse Cubic complexity O(N 3 )
32 Locally Linear Embedding manifold is a topological space which is locally Euclidean. Fit Locally, Think Globally
33 Fit Locally We expect each data point and its neighbours to lie on or close to a locally linear patch of the manifold. Each point can be written as a linear combination of its neighbors. The weights choosen to minimize the reconstruction Error. Derivation on board
34 Important property... The weights that minimize the reconstrucdon errors are invariant to rotadon, rescaling and transladon of the data points. Invariance to transladon is enforced by adding the constraint that the weights sum to one. The same weights that reconstruct the datapoints in D dimensions should reconstruct it in the manifold in d dimensions. The weights characterize the intrinsic geometric properdes of each neighborhood.
35 Think Globally
36 Algorithm (K-NN) Local fipng step (with centering): Consider a point x i Choose its K(i) neighbors η j whose origin is at x i Compute the (sum- to- one) weights w ij which minimizes Ψ i K(i) ( w) = x i w ij η j Contruct neighborhood inner product: C jk = η j,η k Compute the weight ector w i =(w ij ), where 1 is K- vector of ll- one and λ is a regularizadon parameter Then normalize w i to a sum- to- one vector. j =1 2, w ij =1, x i = 0 ( ) 1 1 w i = C + λi j
37 Algorithm (K-NN) Local fipng step (without centering): Consider a point x i Choose its K(i) neighbors x j Compute the (sum- to- one) weights w ij which minimizes Ψ i ( w) = x i w ij x j Contruct neighborhood inner product: C jk = η j,η k Compute the weight ector w i =(w ij ), where K(i) j =1 2, v ik = η k, x i w i = C + v i, v i = ( v ) ik R K(i)
38 Algorithm continued Global embedding step: Construct N- by- N weight matrix W: Compute d- by- N matrix Y which minimizes ( ) = Y i W ij Y j φ Y i N j =1 2 W ij = w ij, j N(i) 0, otherwise = Y(I W ) T (I W )Y T Compute: B = (I W ) T (I W ) Find d+1 bo#om eigenvectors of B, v n,v n- 1,,v n- d Let d- dimensional embedding Y =[ v n- 1,v n- 2, v n- 1 ]
39 Remarks on LLE Searching k- nearest neighbors is of O(kN) W is sparse, kn/n^2=k/n nozeros W might be negadve, addidonal nonnegadve constraint can be imposed B=(I- W) T (I- W) is posidve semi- definite (p.s.d.) Open Problem: exact reconstrucdon condidon?
40
41
42
43 Grolliers Encyclopedia
44 Summary.. ISOMAP Do MDS on the geodesic distance matrix. Global approach LLE Model local neighborhoods as linear a patches and then embed in a lower dimensional manifold. Local approach Might not work for nonconvex manifolds with holes Extensions: Landmark, Conformal & Isometric ISOMAP onconvex manifolds with holes xtensions: Hessian LLE Laplacian Eigenmaps etc Both needs manifold finely sampled
45 Landmark (Sparse) ISOMAP ISOMAP out of the box is not scalable. Two bottlenecks: All pairs shortest path - O(kN 2 log N). MDS eigenvalue calculation on a full NxN matrix - O(N 3 ). For contrast, LLE is limited by a sparse eigenvalue computation - O(dN 2 ). Landmark ISOMAP (L-ISOMAP) Idea: Use n << N landmark points from {x i } and compute a nxn matrix of geodesic distances, D n, from each data point to the landmark points only. Use new procedure Landmark-MDS (LMDS) to find a Euclidean embedding of all the data utilizes idea of triangulation similar to GPS. Savings: L-ISOMAP will have shortest paths calculation of O(knN log N) and LMDS eigenvalue problem of O(n 2 N).
46 Landmark MDS (Restriction) 1. Designate a set of n landmark points. 2. Apply classical MDS to the nxnmatrix n of the squared distances between each landmark point to find a d-dimensional embedding of these n points. Let L k be the dxnmatrix containing the embedded landmark points constructed by utilizing the calculated eigenvectors v i and eigenvalues λ i. λ1 v T 1 λ2 v T 2 L k =. λ d v T d
47 LMDS (Extension) 3. Apply distance-based triangulation to find a d-dimensional embedding of all N points. Let δ 1,..., δ n be vectors of the squared distances from the i-th landmark to all the landmarks and let δ µ be the mean of these vectors. Let δ x be the vector of squared distances between a point x and the landmark points. Then the i-th component of the embedding vector for y x is: y i x = 1 2 v T i λi ( δ x δ µ ) It can be shown that the above embedding of y x is equivalent to projecting onto the first d principal components of the landmarks. 4. Finally, we can optionally choose to run PCA to reorient our axes.
48 Landmark Choice How many landmark points should we choose?... d + 1 landmarks are enough for the triangulation to locate each point uniquely, but heuristics show that a few more is better for stability. Poorly distributed landmarks could lead to foreshortening projection onto the d-dimensional subspace causes a shortening of distances. Good methods are random OR use more expensive MinMax method that for each new landmark added maximizes the minimum distance to the already chosen ones. Either way, running L-ISOMAP in combination with cross-validation techniques would be useful to find a stable embedding.
49 Further exploration yet Hierarchical landmarks: cover- tree Nyström method
50 L-ISOMAP Examples
51 Generative Models in Manifold Learning
52 Conformal & Isometric Embedding
53 Isometric and Conformal Isometric mapping Intrinsically flat manifold Invariants Geodesic distances are reserved. Metric space under geodesic distance. Conformal Embedding Locally isometric upto a scale factor s(y) EsDmate s(y) and rescale. C- Isomap Original data should be uniformly dense
54
55 Linear, Isometric, Conformal If f is a linear isometry f : R d R D then we can simply use PCA or MDS to recover the d significant dimensions Plane. If f is an isometric embedding f : Y R D then provided that data points are sufficiently dense and Y R d is a convex domain we can use ISOMAP to recover the approximate original structure Swiss Roll. If f is a conformal embedding f : Y R D then we must assume the data is uniformly dense in Y and Y R d is a convex domain and then we can successfuly use C-ISOMAP Fish Bowl.
56 Conformal Isomap Idea behind C-ISOMAP: Not only estimate geodesic distances, but also scalar function s(y). Let µ(i) be the mean distance from xi to its k-nn. Each yi and its k-nn occupy a d-dimensional disk of radius r r depends only on d and sampling density. f maps this disk to approximately a d-dimensional disk on M of radius s(y i )r µ(i) s(y i ). µ(i) is a reasonable estimate of s(yi ) since it will be off by a constant factor (uniform density assumption).
57 C-Isomap We replace each edge weight in G by x i x j / µ(i)µ(j). Everything else is the same. Resulting Effect: magnify regions of high density and shrink regions of low density. A similar convergence theorem as given before can be shown about C-ISOMAP assuming that Y is sampled uniformly from a bounded convex region.
58 C-Isomap Example I We will compare LLE, ISOMAP, C-ISOMAP, and MDS on toy datasets. Conformal Fishbowl: Use stereographic projection to project points uniformly distributed in a disk in R 2 onto a sphere with the top removed. Uniform Fishbowl: Points distributed uniformly on the surface of the fishbowl. Offset Fishbowl: Same as conformal fishbowl but points are sampled in Y with a Gaussian offset from center.
59 C-Isomap Example I
60 C-Isomap Example II 2000 face images were randomly generated varying in distance and left-right pose. Each image is a vector in dimensional space. Below shows the four extreme cases. Conformal because changes in orientation at a long distance will have a smaller effect on local pixel distances than the corresponding change at a shorter distance.
61 C-Isomap Example II C-ISOMAP separates the two intrinsic dimensions cleanly. ISOMAP narrows as faces get further away. LLE is highly distorted.
62 Remark
63 Recap and Problems LLE ISOMAP Approach Local Global Isometry Most of the time, covariance distortion Yes Conformal No Guarantees, but sometimes C-ISOMAP Speed Quadratic in N Cubic in N, but L-ISOMAP How do LLE and L-ISOMAP compare in the quality of their output on real world datasets? can we develop a quantitative metric to evaluate them? How much improvement in classification tasks do NLDR techniques really give over traditional dimensionality reduction techniques? Is there some sort of heuristic for choosing k? Possibly could we utilize heirarchical clustering information in constructing a better graph G? Lots of research potential...
64 Reference Tenenbaum, de Silva, and Langford, A Global Geometric Framework for Nonlinear Dimensionality ReducDon. Science 290: , 22 Dec Roweis and Saul, Nonlinear Dimensionality ReducDon by Locally Linear Embedding. Science 290: , 22 Dec M. Bernstein, V. de Silva, J. Langford, and J. Tenenbaum. Graph ApproximaDons to Geodesics on Embedded Manifolds. Technical Report, Department of Psychology, Stanford University, V. de Silva and J.B. Tenenbaum. Global versus local methods in nonlinear dimensionality reducdon. Neural InformaDon Processing Systems 15 (NIPS 2002), pp , V. de Silva and J.B. Tenenbaum. Unsupervised learning of curved manifolds. Nonlinear EsDmaDon and ClassificaDon, V. de Silva and J.B. Tenenbaum. Sparse mulddimensional scaling using landmark points. Available at: h#p://math.stanford.edu/~silva/public/publicadons.html
65 Slides stolen from Epnger, Vikas C. Raykar Vin de Silva.
Global versus local methods in nonlinear dimensionality reduction
Global versus local methods in nonlinear dimensionality reduction Vin de Silva Department of Mathematics, Stanford University, Stanford. CA 94305 silva@math.stanford.edu Joshua B. Tenenbaum Department
Global versus local methods in nonlinear dimensionality reduction
Global versus local methods in nonlinear dimensionality reduction Vin de Silva Department of Mathematics, Stanford University, Stanford. CA 94305 silva@math.stanford.edu Joshua B. Tenenbaum Department
Dimension Reduction CS534
Dimension Reduction CS534 Why dimension reduction? High dimensionality large number of features E.g., documents represented by thousands of words, millions of bigrams Images represented by thousands of
Data fusion and multi-cue data matching using diffusion maps
Data fusion and multi-cue data matching using diffusion maps Stéphane Lafon Collaborators: Raphy Coifman, Andreas Glaser, Yosi Keller, Steven Zucker (Yale University) Part of this work was supported by
Robust Pose Estimation using the SwissRanger SR-3000 Camera
Robust Pose Estimation using the SwissRanger SR- Camera Sigurjón Árni Guðmundsson, Rasmus Larsen and Bjarne K. Ersbøll Technical University of Denmark, Informatics and Mathematical Modelling. Building,
Iterative Non-linear Dimensionality Reduction by Manifold Sculpting
Iterative Non-linear Dimensionality Reduction by Manifold Sculpting Mike Gashler, Dan Ventura, and Tony Martinez Brigham Young University Provo, UT 84604 Abstract Many algorithms have been recently developed
SELECTION OF THE OPTIMAL PARAMETER VALUE FOR THE LOCALLY LINEAR EMBEDDING ALGORITHM. Olga Kouropteva, Oleg Okun and Matti Pietikäinen
SELECTION OF THE OPTIMAL PARAMETER VALUE FOR THE LOCALLY LINEAR EMBEDDING ALGORITHM Olga Kouropteva, Oleg Okun and Matti Pietikäinen Machine Vision Group, Infotech Oulu and Department of Electrical and
Cluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1
Cluster Analysis Mu-Chun Su Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Introduction Cluster analysis is the formal study of algorithms and methods
Selecting Models from Videos for Appearance-Based Face Recognition
Selecting Models from Videos for Appearance-Based Face Recognition Abdenour Hadid and Matti Pietikäinen Machine Vision Group Infotech Oulu and Department of Electrical and Information Engineering P.O.
Dimension Reduction of Image Manifolds
Dimension Reduction of Image Manifolds Arian Maleki Department of Electrical Engineering Stanford University Stanford, CA, 9435, USA E-mail: arianm@stanford.edu I. INTRODUCTION Dimension reduction of datasets
Correspondence. CS 468 Geometry Processing Algorithms. Maks Ovsjanikov
Shape Matching & Correspondence CS 468 Geometry Processing Algorithms Maks Ovsjanikov Wednesday, October 27 th 2010 Overall Goal Given two shapes, find correspondences between them. Overall Goal Given
CIE L*a*b* color model
CIE L*a*b* color model To further strengthen the correlation between the color model and human perception, we apply the following non-linear transformation: with where (X n,y n,z n ) are the tristimulus
RDRToolbox A package for nonlinear dimension reduction with Isomap and LLE.
RDRToolbox A package for nonlinear dimension reduction with Isomap and LLE. Christoph Bartenhagen October 30, 2017 Contents 1 Introduction 1 1.1 Loading the package......................................
Topological estimation using witness complexes. Vin de Silva, Stanford University
Topological estimation using witness complexes, Acknowledgements Gunnar Carlsson (Mathematics, Stanford) principal collaborator Afra Zomorodian (CS/Robotics, Stanford) persistent homology software Josh
Clustering Data That Resides on a Low-Dimensional Manifold in a High-Dimensional Measurement Space. Avinash Kak Purdue University
Clustering Data That Resides on a Low-Dimensional Manifold in a High-Dimensional Measurement Space Avinash Kak Purdue University February 15, 2016 10:37am An RVL Tutorial Presentation Originally presented
CSE 6242 A / CS 4803 DVA. Feb 12, Dimension Reduction. Guest Lecturer: Jaegul Choo
CSE 6242 A / CS 4803 DVA Feb 12, 2013 Dimension Reduction Guest Lecturer: Jaegul Choo CSE 6242 A / CS 4803 DVA Feb 12, 2013 Dimension Reduction Guest Lecturer: Jaegul Choo Data is Too Big To Do Something..
Dimension reduction for hyperspectral imaging using laplacian eigenmaps and randomized principal component analysis:midyear Report
Dimension reduction for hyperspectral imaging using laplacian eigenmaps and randomized principal component analysis:midyear Report Yiran Li yl534@math.umd.edu Advisor: Wojtek Czaja wojtek@math.umd.edu
Definition A metric space is proper if all closed balls are compact. The length pseudo metric of a metric space X is given by.
Chapter 1 Geometry: Nuts and Bolts 1.1 Metric Spaces Definition 1.1.1. A metric space is proper if all closed balls are compact. The length pseudo metric of a metric space X is given by (x, y) inf p. p:x
Image Processing. Image Features
Image Processing Image Features Preliminaries 2 What are Image Features? Anything. What they are used for? Some statements about image fragments (patches) recognition Search for similar patches matching
Data Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University
Data Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University Exploratory data analysis tasks Examine the data, in search of structures
Differential Structure in non-linear Image Embedding Functions
Differential Structure in non-linear Image Embedding Functions Robert Pless Department of Computer Science, Washington University in St. Louis pless@cse.wustl.edu Abstract Many natural image sets are samples
LOCAL LINEAR APPROXIMATION OF PRINCIPAL CURVE PROJECTIONS. Peng Zhang, Esra Ataer-Cansizoglu, Deniz Erdogmus
2012 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 23 26, 2012, SATANDER, SPAIN LOCAL LINEAR APPROXIMATION OF PRINCIPAL CURVE PROJECTIONS Peng Zhang, Esra Ataer-Cansizoglu,
Detection and approximation of linear structures in metric spaces
Stanford - July 12, 2012 MMDS 2012 Detection and approximation of linear structures in metric spaces Frédéric Chazal Geometrica group INRIA Saclay Joint work with M. Aanjaneya, D. Chen, M. Glisse, L. Guibas,
274 Curves on Surfaces, Lecture 5
274 Curves on Surfaces, Lecture 5 Dylan Thurston Notes by Qiaochu Yuan Fall 2012 5 Ideal polygons Previously we discussed three models of the hyperbolic plane: the Poincaré disk, the upper half-plane,
Greedy Routing with Guaranteed Delivery Using Ricci Flow
Greedy Routing with Guaranteed Delivery Using Ricci Flow Jie Gao Stony Brook University Joint work with Rik Sarkar, Xiaotian Yin, Wei Zeng, Feng Luo, Xianfeng David Gu Greedy Routing Assign coordinatesto
Local Linear Embedding. Katelyn Stringer ASTR 689 December 1, 2015
Local Linear Embedding Katelyn Stringer ASTR 689 December 1, 2015 Idea Behind LLE Good at making nonlinear high-dimensional data easier for computers to analyze Example: A high-dimensional surface Think
Edge and local feature detection - 2. Importance of edge detection in computer vision
Edge and local feature detection Gradient based edge detection Edge detection by function fitting Second derivative edge detectors Edge linking and the construction of the chain graph Edge and local feature
Estimating Cluster Overlap on Manifolds and its Application to Neuropsychiatric Disorders
Estimating Cluster Overlap on Manifolds and its Application to Neuropsychiatric Disorders Peng Wang 1, Christian Kohler 2, Ragini Verma 1 1 Department of Radiology University of Pennsylvania Philadelphia,
LEARNING ON STATISTICAL MANIFOLDS FOR CLUSTERING AND VISUALIZATION
LEARNING ON STATISTICAL MANIFOLDS FOR CLUSTERING AND VISUALIZATION Kevin M. Carter 1, Raviv Raich, and Alfred O. Hero III 1 1 Department of EECS, University of Michigan, Ann Arbor, MI 4819 School of EECS,
CAT(0) BOUNDARIES OF TRUNCATED HYPERBOLIC SPACE
CAT(0) BOUNDARIES OF TRUNCATED HYPERBOLIC SPACE KIM RUANE Abstract. We prove that the CAT(0) boundary of a truncated hyperbolic space is homeomorphic to a sphere with disks removed. In dimension three,
Texture Mapping using Surface Flattening via Multi-Dimensional Scaling
Texture Mapping using Surface Flattening via Multi-Dimensional Scaling Gil Zigelman Ron Kimmel Department of Computer Science, Technion, Haifa 32000, Israel and Nahum Kiryati Department of Electrical Engineering
Head Frontal-View Identification Using Extended LLE
Head Frontal-View Identification Using Extended LLE Chao Wang Center for Spoken Language Understanding, Oregon Health and Science University Abstract Automatic head frontal-view identification is challenging
Unsupervised Learning
Networks for Pattern Recognition, 2014 Networks for Single Linkage K-Means Soft DBSCAN PCA Networks for Kohonen Maps Linear Vector Quantization Networks for Problems/Approaches in Machine Learning Supervised
SOLVING PARTIAL DIFFERENTIAL EQUATIONS ON POINT CLOUDS
SOLVING PARTIAL DIFFERENTIAL EQUATIONS ON POINT CLOUDS JIAN LIANG AND HONGKAI ZHAO Abstract. In this paper we present a general framework for solving partial differential equations on manifolds represented
The Pre-Image Problem in Kernel Methods
The Pre-Image Problem in Kernel Methods James Kwok Ivor Tsang Department of Computer Science Hong Kong University of Science and Technology Hong Kong The Pre-Image Problem in Kernel Methods ICML-2003 1
Non-linear CCA and PCA by Alignment of Local Models
Non-linear CCA and PCA by Alignment of Local Models Jakob J. Verbeek, Sam T. Roweis, and Nikos Vlassis Informatics Institute, University of Amsterdam Department of Computer Science,University of Toronto
The Farey Tessellation
The Farey Tessellation Seminar: Geometric Structures on manifolds Mareike Pfeil supervised by Dr. Gye-Seon Lee 15.12.2015 Introduction In this paper, we are going to introduce the Farey tessellation. Since
Réduction de modèles pour des problèmes d optimisation et d identification en calcul de structures
Réduction de modèles pour des problèmes d optimisation et d identification en calcul de structures Piotr Breitkopf Rajan Filomeno Coelho, Huanhuan Gao, Anna Madra, Liang Meng, Guénhaël Le Quilliec, Balaji
LARGE dimensionality is a significant problem for many
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS PART B: CYBERNETICS 1 Manifold Learning by Graduated Optimization Michael Gashler, Dan Ventura, and Tony Martinez Abstract We present an algorithm for
Non-Local Estimation of Manifold Structure
Non-Local Estimation of Manifold Structure Yoshua Bengio, Martin Monperrus and Hugo Larochelle Département d Informatique et Recherche Opérationnelle Centre de Recherches Mathématiques Université de Montréal
Data Preprocessing. Javier Béjar AMLT /2017 CS - MAI. (CS - MAI) Data Preprocessing AMLT / / 71 BY: $\
Data Preprocessing S - MAI AMLT - 2016/2017 (S - MAI) Data Preprocessing AMLT - 2016/2017 1 / 71 Outline 1 Introduction Data Representation 2 Data Preprocessing Outliers Missing Values Normalization Discretization
CPSC 340: Machine Learning and Data Mining. Kernel Trick Fall 2017
CPSC 340: Machine Learning and Data Mining Kernel Trick Fall 2017 Admin Assignment 3: Due Friday. Midterm: Can view your exam during instructor office hours or after class this week. Digression: the other
Extended Isomap for Pattern Classification
From: AAAI- Proceedings. Copyright, AAAI (www.aaai.org). All rights reserved. Extended for Pattern Classification Ming-Hsuan Yang Honda Fundamental Research Labs Mountain View, CA 944 myang@hra.com Abstract
Generalized Principal Component Analysis CVPR 2007
Generalized Principal Component Analysis Tutorial @ CVPR 2007 Yi Ma ECE Department University of Illinois Urbana Champaign René Vidal Center for Imaging Science Institute for Computational Medicine Johns
Computer vision: models, learning and inference. Chapter 13 Image preprocessing and feature extraction
Computer vision: models, learning and inference Chapter 13 Image preprocessing and feature extraction Preprocessing The goal of pre-processing is to try to reduce unwanted variation in image due to lighting,
Towards Multi-scale Heat Kernel Signatures for Point Cloud Models of Engineering Artifacts
Towards Multi-scale Heat Kernel Signatures for Point Cloud Models of Engineering Artifacts Reed M. Williams and Horea T. Ilieş Department of Mechanical Engineering University of Connecticut Storrs, CT
A Stochastic Optimization Approach for Unsupervised Kernel Regression
A Stochastic Optimization Approach for Unsupervised Kernel Regression Oliver Kramer Institute of Structural Mechanics Bauhaus-University Weimar oliver.kramer@uni-weimar.de Fabian Gieseke Institute of Structural
Surface Reconstruction with MLS
Surface Reconstruction with MLS Tobias Martin CS7960, Spring 2006, Feb 23 Literature An Adaptive MLS Surface for Reconstruction with Guarantees, T. K. Dey and J. Sun A Sampling Theorem for MLS Surfaces,
Subdivision Surfaces
Subdivision Surfaces 1 Geometric Modeling Sometimes need more than polygon meshes Smooth surfaces Traditional geometric modeling used NURBS Non uniform rational B-Spline Demo 2 Problems with NURBS A single
Indian Institute of Technology Kanpur. Visuomotor Learning Using Image Manifolds: ST-GK Problem
Indian Institute of Technology Kanpur Introduction to Cognitive Science SE367A Visuomotor Learning Using Image Manifolds: ST-GK Problem Author: Anurag Misra Department of Computer Science and Engineering
A Taxonomy of Semi-Supervised Learning Algorithms
A Taxonomy of Semi-Supervised Learning Algorithms Olivier Chapelle Max Planck Institute for Biological Cybernetics December 2005 Outline 1 Introduction 2 Generative models 3 Low density separation 4 Graph
Hyperbolic structures and triangulations
CHAPTER Hyperbolic structures and triangulations In chapter 3, we learned that hyperbolic structures lead to developing maps and holonomy, and that the developing map is a covering map if and only if the
Inverse and Implicit functions
CHAPTER 3 Inverse and Implicit functions. Inverse Functions and Coordinate Changes Let U R d be a domain. Theorem. (Inverse function theorem). If ϕ : U R d is differentiable at a and Dϕ a is invertible,
Non-Local Estimation of Manifold Structure
Neural Computation Manuscript #3171 Non-Local Estimation of Manifold Structure Yoshua Bengio, Martin Monperrus and Hugo Larochelle Département d Informatique et Recherche Opérationnelle Centre de Recherches
Shape spaces. Shape usually defined explicitly to be residual: independent of size.
Shape spaces Before define a shape space, must define shape. Numerous definitions of shape in relation to size. Many definitions of size (which we will explicitly define later). Shape usually defined explicitly
Graphs associated to CAT(0) cube complexes
Graphs associated to CAT(0) cube complexes Mark Hagen McGill University Cornell Topology Seminar, 15 November 2011 Outline Background on CAT(0) cube complexes The contact graph: a combinatorial invariant
Semi-supervised Learning with Density Based Distances
Semi-supervised Learning with Density Based Distances Avleen S. Bijral Toyota Technological Institute Chicago, IL 6637 Nathan Ratliff Google Inc. Pittsburg, PA 526 Nathan Srebro Toyota Technological Institute
THE recovery of low-dimensional structures in data sets. Out-of-sample generalizations for supervised manifold learning for classification
1 Out-of-sample generalizations for supervised manifold learning for classification Elif Vural and Christine Guillemot arxiv:152.241v1 [cs.cv] 9 Feb 215 Abstract Supervised manifold learning methods for
Classification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University
Classification Vladimir Curic Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University Outline An overview on classification Basics of classification How to choose appropriate
The exam is closed book, closed notes except your one-page (two-sided) cheat sheet.
CS 189 Spring 2015 Introduction to Machine Learning Final You have 2 hours 50 minutes for the exam. The exam is closed book, closed notes except your one-page (two-sided) cheat sheet. No calculators or
Surface Parameterization
Surface Parameterization A Tutorial and Survey Michael Floater and Kai Hormann Presented by Afra Zomorodian CS 468 10/19/5 1 Problem 1-1 mapping from domain to surface Original application: Texture mapping
Application of Spectral Clustering Algorithm
1/27 Application of Spectral Clustering Algorithm Danielle Middlebrooks dmiddle1@math.umd.edu Advisor: Kasso Okoudjou kasso@umd.edu Department of Mathematics University of Maryland- College Park Advance
Spectral Clustering. Presented by Eldad Rubinstein Based on a Tutorial by Ulrike von Luxburg TAU Big Data Processing Seminar December 14, 2014
Spectral Clustering Presented by Eldad Rubinstein Based on a Tutorial by Ulrike von Luxburg TAU Big Data Processing Seminar December 14, 2014 What are we going to talk about? Introduction Clustering and
Two Connections between Combinatorial and Differential Geometry
Two Connections between Combinatorial and Differential Geometry John M. Sullivan Institut für Mathematik, Technische Universität Berlin Berlin Mathematical School DFG Research Group Polyhedral Surfaces
An Efficient Algorithm for Local Distance Metric Learning
An Efficient Algorithm for Local Distance Metric Learning Liu Yang and Rong Jin Michigan State University Dept. of Computer Science & Engineering East Lansing, MI 48824 {yangliu1, rongjin}@cse.msu.edu
Clustering. SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic
Clustering SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic Clustering is one of the fundamental and ubiquitous tasks in exploratory data analysis a first intuition about the
A GENTLE INTRODUCTION TO THE BASIC CONCEPTS OF SHAPE SPACE AND SHAPE STATISTICS
A GENTLE INTRODUCTION TO THE BASIC CONCEPTS OF SHAPE SPACE AND SHAPE STATISTICS HEMANT D. TAGARE. Introduction. Shape is a prominent visual feature in many images. Unfortunately, the mathematical theory
WITH the wide usage of information technology in almost. Supervised Nonlinear Dimensionality Reduction for Visualization and Classification
1098 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS PART B: CYBERNETICS, VOL. 35, NO. 6, DECEMBER 2005 Supervised Nonlinear Dimensionality Reduction for Visualization and Classification Xin Geng, De-Chuan
Hyperspectral image segmentation using spatial-spectral graphs
Hyperspectral image segmentation using spatial-spectral graphs David B. Gillis* and Jeffrey H. Bowles Naval Research Laboratory, Remote Sensing Division, Washington, DC 20375 ABSTRACT Spectral graph theory
Mapping a manifold of perceptual observations
Mapping a manifold of perceptual observations Joshua. Tenenbaum Department of rain and Cognitive Sciences Massachusetts Institute of Technology, Cambridge, M 2139 jbt@psyche.mit.edu bstract Nonlinear dimensionality
Semi-supervised Regression using Hessian Energy with an Application to Semi-supervised Dimensionality Reduction
Semi-supervised Regression using Hessian Energy with an Application to Semi-supervised Dimensionality Reduction Kwang In Kim, Florian Steinke,3, and Matthias Hein Department of Computer Science, Saarland
Nonlinear dimensionality reduction of large datasets for data exploration
Data Mining VII: Data, Text and Web Mining and their Business Applications 3 Nonlinear dimensionality reduction of large datasets for data exploration V. Tomenko & V. Popov Wessex Institute of Technology,
Clustering and Visualisation of Data
Clustering and Visualisation of Data Hiroshi Shimodaira January-March 28 Cluster analysis aims to partition a data set into meaningful or useful groups, based on distances between data points. In some
Lecture 2 The k-means clustering problem
CSE 29: Unsupervised learning Spring 2008 Lecture 2 The -means clustering problem 2. The -means cost function Last time we saw the -center problem, in which the input is a set S of data points and the
Topic: Orientation, Surfaces, and Euler characteristic
Topic: Orientation, Surfaces, and Euler characteristic The material in these notes is motivated by Chapter 2 of Cromwell. A source I used for smooth manifolds is do Carmo s Riemannian Geometry. Ideas of
Surface Reconstruction from Unorganized Points
Survey of Methods in Computer Graphics: Surface Reconstruction from Unorganized Points H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, W. Stuetzle SIGGRAPH 1992. Article and Additional Material at: http://research.microsoft.com/en-us/um/people/hoppe/proj/recon/
Introduction to geometry
1 2 Manifolds A topological space in which every point has a neighborhood homeomorphic to (topological disc) is called an n-dimensional (or n-) manifold Introduction to geometry The German way 2-manifold
Planes Intersecting Cones: Static Hypertext Version
Page 1 of 12 Planes Intersecting Cones: Static Hypertext Version On this page, we develop some of the details of the plane-slicing-cone picture discussed in the introduction. The relationship between the
Research in Computational Differential Geomet
Research in Computational Differential Geometry November 5, 2014 Approximations Often we have a series of approximations which we think are getting close to looking like some shape. Approximations Often
PATTERN CLASSIFICATION AND SCENE ANALYSIS
PATTERN CLASSIFICATION AND SCENE ANALYSIS RICHARD O. DUDA PETER E. HART Stanford Research Institute, Menlo Park, California A WILEY-INTERSCIENCE PUBLICATION JOHN WILEY & SONS New York Chichester Brisbane
arxiv:math/ v3 [math.dg] 23 Jul 2007
THE INTRINSIC GEOMETRY OF A JORDAN DOMAIN arxiv:math/0512622v3 [math.dg] 23 Jul 2007 RICHARD L. BISHOP Abstract. For a Jordan domain in the plane the length metric space of points connected to an interior
5. THE ISOPERIMETRIC PROBLEM
Math 501 - Differential Geometry Herman Gluck March 1, 2012 5. THE ISOPERIMETRIC PROBLEM Theorem. Let C be a simple closed curve in the plane with length L and bounding a region of area A. Then L 2 4 A,
A non-negative representation learning algorithm for selecting neighbors
Mach Learn (2016) 102:133 153 DOI 10.1007/s10994-015-5501-4 A non-negative representation learning algorithm for selecting neighbors Lili Li 1 Jiancheng Lv 1 Zhang Yi 1 Received: 8 August 2014 / Accepted:
Supervised vs unsupervised clustering
Classification Supervised vs unsupervised clustering Cluster analysis: Classes are not known a- priori. Classification: Classes are defined a-priori Sometimes called supervised clustering Extract useful
Lecture 2. Topology of Sets in R n. August 27, 2008
Lecture 2 Topology of Sets in R n August 27, 2008 Outline Vectors, Matrices, Norms, Convergence Open and Closed Sets Special Sets: Subspace, Affine Set, Cone, Convex Set Special Convex Sets: Hyperplane,
Shape Classification and Cell Movement in 3D Matrix Tutorial (Part I)
Shape Classification and Cell Movement in 3D Matrix Tutorial (Part I) Fred Park UCI icamp 2011 Outline 1. Motivation and Shape Definition 2. Shape Descriptors 3. Classification 4. Applications: Shape Matching,
Outline. Advanced Digital Image Processing and Others. Importance of Segmentation (Cont.) Importance of Segmentation
Advanced Digital Image Processing and Others Xiaojun Qi -- REU Site Program in CVIP (7 Summer) Outline Segmentation Strategies and Data Structures Algorithms Overview K-Means Algorithm Hidden Markov Model
Algebraic Geometry of Segmentation and Tracking
Ma191b Winter 2017 Geometry of Neuroscience Geometry of lines in 3-space and Segmentation and Tracking This lecture is based on the papers: Reference: Marco Pellegrini, Ray shooting and lines in space.
MultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A
MultiDimensional Signal Processing Master Degree in Ingegneria delle Telecomunicazioni A.A. 205-206 Pietro Guccione, PhD DEI - DIPARTIMENTO DI INGEGNERIA ELETTRICA E DELL INFORMAZIONE POLITECNICO DI BARI
Advanced Topics In Machine Learning Project Report : Low Dimensional Embedding of a Pose Collection Fabian Prada
Advanced Topics In Machine Learning Project Report : Low Dimensional Embedding of a Pose Collection Fabian Prada 1 Introduction In this project we present an overview of (1) low dimensional embedding,
Manifold Alignment. Chang Wang, Peter Krafft, and Sridhar Mahadevan
Manifold Alignment Chang Wang, Peter Krafft, and Sridhar Mahadevan April 1, 211 2 Contents 5 Manifold Alignment 5 5.1 Introduction.................................... 5 5.1.1 Problem Statement............................
An Introduction to Belyi Surfaces
An Introduction to Belyi Surfaces Matthew Stevenson December 16, 2013 We outline the basic theory of Belyi surfaces, up to Belyi s theorem (1979, [1]), which characterizes these spaces as precisely those
Machine Learning: Think Big and Parallel
Day 1 Inderjit S. Dhillon Dept of Computer Science UT Austin CS395T: Topics in Multicore Programming Oct 1, 2013 Outline Scikit-learn: Machine Learning in Python Supervised Learning day1 Regression: Least
Machine Learning : Clustering, Self-Organizing Maps
Machine Learning Clustering, Self-Organizing Maps 12/12/2013 Machine Learning : Clustering, Self-Organizing Maps Clustering The task: partition a set of objects into meaningful subsets (clusters). The
LTI Thesis Defense: Riemannian Geometry and Statistical Machine Learning
Outline LTI Thesis Defense: Riemannian Geometry and Statistical Machine Learning Guy Lebanon Motivation Introduction Previous Work Committee:John Lafferty Geoff Gordon, Michael I. Jordan, Larry Wasserman
3 Nonlinear Regression
CSC 4 / CSC D / CSC C 3 Sometimes linear models are not sufficient to capture the real-world phenomena, and thus nonlinear models are necessary. In regression, all such models will have the same basic
Diffusion Maps and Topological Data Analysis
Diffusion Maps and Topological Data Analysis Melissa R. McGuirl McGuirl (Brown University) Diffusion Maps and Topological Data Analysis 1 / 19 Introduction OVERVIEW Topological Data Analysis The use of
Mathematical Programming and Research Methods (Part II)
Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types
Biometrics Technology: Image Processing & Pattern Recognition (by Dr. Dickson Tong)
Biometrics Technology: Image Processing & Pattern Recognition (by Dr. Dickson Tong) References: [1] http://homepages.inf.ed.ac.uk/rbf/hipr2/index.htm [2] http://www.cs.wisc.edu/~dyer/cs540/notes/vision.html
Kernels and representation
Kernels and representation Corso di AA, anno 2017/18, Padova Fabio Aiolli 20 Dicembre 2017 Fabio Aiolli Kernels and representation 20 Dicembre 2017 1 / 19 (Hierarchical) Representation Learning Hierarchical