Non-linear dimension reduction

Transcription

1 Sta306b May 23, 2011 Dimension Reduction: 1 Non-linear dimension reduction ISOMAP: Tenenbaum, de Silva & Langford (2000) Local linear embedding: Roweis & Saul (2000) Local MDS: Chen (2006) all three methods try to map points on a high-dimensional non-linear manifold to a lower dimensional set of coordinates useful for problems where signal-to-noise ratio is very high (e.g. a physical system). May not be useful for observational data

2 Sta306b May 23, 2011 Dimension Reduction: 2 Swiss roll

3 Sta306b May 23, 2011 Dimension Reduction: 3 The Swiss roll data set, illustrating how Isomap exploits geodesic paths for nonlinear dimensionality reduction. (A) For two arbitrary points (circled) on a nonlinear manifold, their Euclidean distance in the high-dimensional input space (length of dashed line) may not accurately reflect their intrinsic similarity, as measured by geodesic distance along the low-dimensional manifold (length of solid curve). (B) The neighborhood graph G constructed in step one of Isomap (with K = 7 and N = 1000 data points) allows an approximation (red segments) to the true geodesic path to be computed efficiently in step two, as the shortest path in G. (C) The two-dimensional embedding recovered by Isomap in step three, which best preserves the shortest path distances in the neighborhood graph (overlaid). Straight lines in the embedding (blue) now represent simpler and cleaner approximations to the true geodesic paths than do the corresponding graph paths (red)

4 Sta306b May 23, 2011 Dimension Reduction: 4 ISOMAP we start with many data points in high dimensional space, lying near some manifold For each data point i we find the points j on manifold within some Euclidean distance d X (i,j) ǫ We construct a graph on the manifold with an edge between i and j if d X (i,j) ǫ We find the shortest path d G (i,j) between points i and j on the graph. Finally, we apply classical MDS to the distances d G (i,j)

5 Sta306b May 23, 2011 Dimension Reduction: 5 Hand position example

6 Sta306b May 23, 2011 Dimension Reduction: 6

7 Sta306b May 23, 2011 Dimension Reduction: 7 Isomap (K=6) applied to N=2000 images (64 pixels by 64 pixels) of a hand in different configurations. The images were generated by making a series of opening and closing movements of the hand at different wrist orientations, designed to give rise to a two-dimensional manifold. The images were treated as 4096-dimensional vectors, with input-space distances defined in the Euclidean metric. As shown in Fig. 2C of the paper, Isomap correctly detects two clearly significant dimensions, plus several weak dimensions of noise; PCA and MDS do not detect the correct dimensionality and suggest a much higher level of noise. The recovered coordinate axes map approximately onto the distinct underlying degrees of freedom: wrist rotation (x axis) and finger extension (y axis).

8 Sta306b May 23, 2011 Dimension Reduction: 8 LLE 1. For each data point i in p dims, we find the K- nearest neighbors N(i) in Euclidean distance 2. compute a kind of local principal component plane to the points in the neighborhood, minimizing x i W ij x j 2 i j N(i) over weights W ij satisfying j W ij = 1. [weights not assumed to be non-negative]? W ij is the contribution of point j to the reconstruction of point i. 3. Finally, we find points y i in a lower dimensional space to minimize y i W ij y j 2 i j with W ij fixed.

9 Sta306b May 23, 2011 Dimension Reduction: 9 Details of step 3 Minimize tr[(y WY) T (Y WY)] = tr[y(i W) T (I W)Y] W is N N; Y is N d, for some small d < p. solutions Y are the bottom eigenvectors of M = (I W) T (I W). They also assume 1 T Y = 0 (i.e. they are centered at the origin), and (1/N)Y T Y = I, the identity matrix in d dimensions. This means they discard the smallest eigenvector of M and keep the next d.

10 Sta306b May 23, 2011 Dimension Reduction: 10

11 Sta306b May 23, 2011 Dimension Reduction: 11 Swissroll

12 Sta306b May 23, 2011 Dimension Reduction: 12 The problem of nonlinear dimensionality reduction, as illustrated (10) for three-dimensional data (B) sampled from a two-dimensional manifold (A). An unsupervised learning algorithm must discover the global internal coordinates of the manifold without signals that explicitly indicate how the data should be embedded in two dimensions. The color coding illustrates the neighborhood-preserving mapping discovered by LLE; black outlines in (B) and (C) show the neighborhood of a single point. Unlike LLE, projections of the data by principal component analysis (PCA) (28) or classical MDS (2) map faraway data points to nearby points in the plane, failing to identify the underlying structure of the manifold. Note that mixture models for local dimensionality reduction (29), which cluster the data and perform PCA within each cluster, do not address the problem considered here: namely, how to map high-dimensional data into a single global coordinate system of lower dimensionality.

13 Sta306b May 23, 2011 Dimension Reduction: 13 Faces

14 Sta306b May 23, 2011 Dimension Reduction: 14 Images of faces (11) mapped into the embedding space described by the first two coordinates of LLE. Representative faces are shown next to circled points in different parts of the space. The bottom images correspond to points along the top-right path (linked by solid line), illustrating one particular mode of variability in pose and expression.

15 Sta306b May 23, 2011 Dimension Reduction: 15 Local multidimensional scaling Local MDS (Chen 2006) takes the simplest and arguably the most direct approach. We define N to be the symmetric set of nearby pairs of points; specifically a pair (i,i ) is in N if point i is among the K-nearest neighbors of i, or vice-versa. Then we construct the stress function S(z 1,z 2,...,z N ) = (d ii z i z i ) 2 (i,i ) N + (i,i )/ N w (D z i z i ) 2. (1)

16 Sta306b May 23, 2011 Dimension Reduction: 16 Local multidimensional scaling -ctd Here D is some large constant and w is a weight. The idea is that points that are not neighbors are considered to be very far apart; such pairs are given a small weight w so that they don t dominate the overall stress function. To simplify the expression, we take w 1/D, and let D. Expanding (1), this gives S(z 1,z 2,...,z N ) = (d ii z i z i ) 2 t z i z i, where t = 2wD. (i,i ) N (i,i )/ N (2)

17 Sta306b May 23, 2011 Dimension Reduction: 17 Local multidimensional scaling -ctd The first term in (2) tries to preserve local structure in the data, while the second term encourages image pairs (i,i ) for non-neighbors to be farther apart in the representation. Local MDS minimizes the stress function (2) over z i, for fixed values of the number of neighbors K and the tuning parameter t. In experiments reported in Chen (2006), local MDS shows superior performance, as compared to ISOMAP and LLE. They also demonstrate the usefulness of local MDS for graph layout. There are also close connections between the methods discussed here, spectral clustering and kernel PCA.

18 Sta306b May 23, 2011 Dimension Reduction: 17-1 References Chen, L. (2006), Local Multidimensional Scaling for Nonlinear Dimension Reduction, Graph Layout and Proximity Analysis, PhD thesis, University of Pennsylvania. Roweis, S. T. & Saul, L. K. (2000), Locally linear embedding, Science 290, Tenenbaum, J. B., de Silva, V. & Langford, J. C. (2000), A global geometric framework for nonlinear dimensionality reduction, Science 290,