MiPPS: A Generative Model for Multi-Manifold Clustering

Transcription

1 Manifold Learning and its Applications: Papers fro the AAAI Fall Syposiu (FS-9-) MiPPS: A Generative Model for Multi-Manifold Clustering Oluwasani Koyejo and Joydeep Ghosh Electrical and Coputer Engineering Dept., University Of Texas, Austin sani.k@ail.utexas.edu, ghosh@ece.utexas.edu Abstract We propose a generative odel for high diensional data consisting of intrinsically low diensional clusters that are noisily sapled. The proposed odel is a ixture of probabilistic principal surfaces (MiPPS) optiized using expectation axiization. We use a Bayesian prior on the odel paraeters to axiize the corresponding arginal likelihood. We also show epirically that this optiization can be biased towards a good local optiu by using our prior intuition to guide the initialization phase. The proposed unsupervised algorith naturally handles cases where the data lies on ultiple connected coponents of a single anifold and where the coponent anifolds intersect. In addition to clustering, we learn a functional odel for the underlying structure of each coponent cluster as a paraeterized hyper-surface in abient noise. This odel is used to learn a global ebedding that we use for visualization of the entire dataset. We deonstrate the perforance of MiPPS in separating and visualizing land cover types in a hyperspectral dataset.. Introduction Consider a data set consisting of iages of various objects fro all possible orientations. With all other conditions fixed, there are two degrees of freedo for a given object in this dataset; the elevation and rotation of its corresponding iages. This iplies that the set of all iages in our database is likely to be well represented as soe two diensional non-linear anifold in the iage space with ultiple connected coponents corresponding to each object. This property is not unique to object pose iages. Other coon exaples include iages of faces, handwritten digits and shapes. Now, suppose one would like to partition the iage data set in an unsupervised anner, for exaple, to separate the data by object with each cluster containing all poses of the object in question. Special care should be taken when the data is generated fro one or ore non-linear anifolds. Coon ethods for data clustering such as k-eans or a ixture of Gaussians are unlikely to find the correct partitions as these ethods assue that the data clusters are convex; an assuption that is often untrue in coplicated Copyright c 9, Association for the Advanceent of Artificial Intelligence ( All rights reserved. datasets. Further, one ight be interested in learning a paraetric odel of each anifold for copression, diensionality reduction, visualization, generating novel saples or other post-processing. This otivates studying special paraetric ethods for ulti-anifold clustering. When the data saples are generated fro separated clusters on a connected anifold, ebedding techniques such as ISOMAP and LLE can be used to unwrap the anifold; creating a low diensional ebedding where the data can be clustered ore easily (Yankov and Keogh ). The quality of the ebedding and subsequent clustering found by these ethods is very sensitive to noise and paraeter selection. Further, ISOMAP and LLE will fail to find to find an ebedding when there is a large separation between the clusters. Statistical ebedding and data visualization approaches such as the co-ordinated ixture of probabilistic principal coponent analyzers (PPCA) (Verbeek, Vlassis, and Kröse ) and the generative topographic apping (GTM) (Bishop, Svensen, and Willias 998b) are ore robust to noise. In both ethods, the data can be apped to a low diensional latent space. The apped points can then be partitioned to estiate the data clusters. This technique loosely corresponds to a generative odel. However, the resulting heuristic does not necessarily optiize the odel. Tino and Nabney describe a supervised visualization odel based on hierarchical ixtures of GTM (Tino and Nabney ). At each layer, the user initializes the local GTM using interesting points. However, the odel is not designed to learn data clusters in an unsupervised anner. Further, our experients suggest that our base odel is superior to the GTM as a paraetric non-linear anifold odel in ters of odelling and clustering perforance. The algorith ost related to ours is the k-anifolds algorith proposed by (Souvenir ) for estiating and clustering data generated fro intersecting non-linear anifolds. The algorith uses approxiate geodesic distances siilar to ISOMAP, then a node-weighted MDS apping is used to estiate local ebeddings. These local ebeddings are used to update weights for each data point proportional to the distance between the point and the estiated anifold. The weights and the ebeddings are coputed iter- This failure ode is addressed to soe extent by (Hadid and Pietikinen ) and others. 8

2 atively until convergence. The estiation of geodesic distances fails when the clusters are widely separated. For this reason, k-anifolds is priarily otivated for intersecting anifolds. In this paper, we focus on the non-linear anifold case. For this reason, we will not discuss linear ultianifold ethods. A good odel for a dataset generated fro a connected low diensional anifold is the principal surface (Hastie 98); a non-paraetric non-linear anifold that passes through the iddle of the data. The principal surface can also be described as a non-linear generalization of the principal subspace. However, coputing principal surfaces of ore than one diension is coputationally expensive. To address this difficulty, (Chang and Ghosh ) proposed the probabilistic principal surface (PPS); the basis of our odel. In this paper, we seek a principled algorith for siultaneously learning a soft partition of high diensional data and a local paraetric odel for the clusters where each cluster is a low diensional non-linear anifold. In our proposed odel (MiPPS), each cluster is described using a PPS and the entire dataset is odeled as a probabilistic ixture of these coponents. By analyzing the resulting coplete log-likelihood, we learn soft cluster assignents and paraetric odels for each cluster.. Model Description Our odel is based on the assuption that the generating anifold of each cluster has a diensionality Q D where D is the diensionality of the data space Y. We assue that this anifold can be described as the iage of a sooth function f (x) that aps a latent space X Y. This assuption is valid for a large class of anifolds. A new saple is generated in four steps: One of the M clusters is selected with a probability p() = ν. A saple x is generated fro the selected latent space X where x p(x ). The latent saple is apped to the data space via soe non-linear sooth function f (x). The resulting data saple is corrupted by noise ɛ. Using zero ean Gaussian noise, the distribution of each data saple y n given a cluster selection, the corresponding odel paraeters θ and a latent saple x is: p(y n x, θ ) = N (y n f (x), Σ (x)), () where N (y µ, Σ) represents a Gaussian distribution with ean µ and covariance Σ. The PPS uses a radial basis function (RBF) network f (x) = W φ(x) to define the apping between the latent space and the data space. The weight atrix, W has diension D L and φ(x) = [ φ (x) φ L (x)] T is a vector of L latent basis functions: φ l (x) : R Q R. We use isotropic Gaussian radial basis functions; φ l (x) = exp( x c l σ ) given soe centroid c l and a length scale σ which we fix for all l. The topographic constraints enforced by this odel are discussed in (Bishop, Svensen, and Willias 998b). The latent variable distribution p(x ) ust be propagated to the data space through the non-linear apping f (x) to describe the data distribution. The resulting data distribution is coputed by arginalizing () over the latent space. Unfortunately, for general non-linear appings, this integration is intractable. One solution is a discrete approxiation of the integration by sapling over a unifor grid of K points in the latent space. The resulting distribution is a constrained ixture: K p(y n θ ) = p(y n x k, θ )p(x k ). () k= The approxiation can be coputed to arbitrary accuracy by increasing the nuber of latent points with a corresponding increase in the coputational cost. The PPS odel separates the noise variance into subspaces parallel and orthogonal to the data anifold with variances given by a and b respectively, where: a = α β b = D α Q β (D Q ). The paraeter α (, D/Q ) controls the aount of anifold alignent in the noise covariance while β controls the noise variance. When α (, ) the noise has a higher variance orthogonal to the anifold. As α, the odel satisfies the self consistency property for principal surfaces (Chang and Ghosh ). When α (, D/Q ), the noise has a higher variance parallel to the anifold, corresponding to the anifold aligned GTM odel (Bishop, Svensen, and Willias 998a). This is useful when one suspects that uch of the sapling noise is tangential to the anifold. Finally, when α =, the noise variance is isotropic and the GTM with Σ (x) = β I is recovered. Chang and Ghosh showed that the PPS learns a ore representative odel than the GTM in any cases. The subspaces parallel and orthogonal to the anifold are given by the orthonoral atrices E // (x) and E (x) respectively. These can easily be coputed using the partial derivatives of the the anifold apping with respect to each latent diension: [ f (x) E // (x) = f ] (x) = W T (x), x x Q E (x) = [E // (x)] = W O (x), () where T (x) = [ φ(x) x φ(x) x Q ] R L Q. The atrices T (x) and O (x) are constant and only need to be coputed at initialization. Further, because [T (x) O (x)] are a basis for R D, only T (x) is required to copletely describe the odel. Unfortunately, neither the rows of W nor the coluns of T (x) are guaranteed to be orthonoral and so a further orthonoralization step such as 9

3 the Gra-Schidt procedure ust be applied to E // (x). Now, the oriented covariance can be coputed as: Σ (x) = a E // (x)e T // (x) + b E (x)e T (x). () Note that the total noise variance easured as the su of the eigenvalues of Σ (x) reains constant equal to D β regardless of varying values of α or the local covariance orientation. For the reainder of this paper, we assue that all the data points are independent and identically distributed. The corresponding log likelihood is then a product over the likelihood of each data point: L D (Θ) = ln = n N p(y n X, Θ)p(X Θ) n= ln,k p(, x k )N (y n W φ(x k ), Σ (x k )), (5) where Θ = {θ } M =. This assuption can be relaxed to odel dependent realizations such as a tie series. We also introduce a Bayesian prior over the weight paraeter W. For siplicity, we use a radially syetric Gaussian prior of the for: p(w {λ }) = p(w λ ) = M p(w λ )δ(i ), where i= ( λ π ) MD/ { exp λ } w. () λ is a hyper-paraeter and w is the vectorized for of W (i.e. the atrix stacked colun-wise into a single vector). Intuitively, this eans we place an independent prior over each W. This prior enforces our soothness assuptions on the weight atrix by effectively regularizing the atrix nor. In suary, the full set of odel paraeters are: {θ } = {W, λ, α, β, ν }.. Optiizing the Model Paraeters The for of the apping and the likelihood cost function suggest optiization using the expectation axiization (EM) optiization technique; a co-ordinate ascent algorith for axiizing data likelihood (Depster, Laird, and Rubin 977). The optiization proceeds by alternately coputing the expected coplete log-likelihood (E-step) then axiizing the expected coplete log-likelihood with respect to the paraeter selections (M-step) till convergence. The EM algorith is guaranteed to increase the objective function at each iteration unless it is at a local optiu. E-step For the E step, the expectation of the coplete loglikelihood is coputed with respect to the distribution of the hidden variables given the data. Given a current paraeter estiate Θ, the expected log-likelihood is given by, L c (Θ) = L D (Θ) + L W (Θ) where L D is the data likelihood and L W is the weight paraeter log-likelihood given by the log of (). L D (Θ) can be expressed as: r,k n {ln p(y n θ, x k ) + ln ν + ln p(x k )}, (7) n,,k where r,k n is posterior probability (also called the responsibility) that the data saple y n is generated fro cluster and latent variable x k. Using Bayes theore, r n,k = p(, x k y n ) = p(, x k )p(y n θ, x k ),k p(, x k )p(y n θ, x k ). (8) In order to copute these probabilities, the deterinant and inverse of the corresponding Gaussian covariances ust be coputed. The PPS odel siplifies this coputation ((Chang and Ghosh ), Proposition ): Σ (x) = a Q b D Q Σ (x) = I (a b ) E // (x)e T // b b a (x). Hence, the cost of coputing Σ (x) is reduced to a single product, while the cost of coputing Σ (x) is reduced fro O(D ) to O(Q D ). M-step The M-step involves coputing the paraeter values Θ that axiize the expected log-likelihood. This axiization generally involves the derivatives of the expected loglikelihood with respect to each of the paraeters. We discuss the paraeters in the order that they are updated. Updating ν : First, differentiating the log likelihood with respect to ν and setting it to zero, ν is updated as: ν = r n N,k, (9) Updating p(x k ) (Optional): Differentiating the log likelihood with respect to p(x k ) and setting it to zero: p(x k ) = r n N,k. () In our ipleentation, we assue that the anifold is uniforly sapled. For this reason, we uniforly saple the latent space as well by fixing p(x k ) = K without update. This helps to avoid overfitting. Updating W: We approxiate W as the weight atrix that locally axiizes the coplete log likelihood. The derivative L C(Θ) W is: { [ r,k n I a ] b W Ψ k W T q n b b a,kφ(x k ) T + a b b a n } [ q n,k q n,kw T Ψ ] k λ W, ()

4 where q n,k = y n W φ(x k ), Ψ k = T (x k )T (x k ) T. This derivative is a non-linear function of the weight atrix. For this reason, the updated weight atrix cannot be coputed in closed for. Here we provide two approxiations as suggested by (Chang and Ghosh ). The first option is to iteratively update W using steepest ascent with learning rate η to find a local axiu. At each iteration i, W i+ = W i + η L C(Θ) W. () This forulation fits into the generalized EM (GEM) fraework, and convergence to a local optiu is guaranteed (Depster, Laird, and Rubin 977). However, there is a high coputational cost involved. The second option is to approxiate the weighting atrix in the M-step by the weighting atrix obtained using a GTM i.e. assuing that the covariance: Σ (x) = β I for the weight update. Setting α =, () siplifies considerably to: r,k n { β [W φ(x k ) y n ] φ(x k ) T } λ W. () This can be solved for a fixed point to obtain the updated weight atrices {W} M = by least squares: ( Φ T G Φ + λ β I ) W T = ΦR Y. () where Y is the N D data atrix with each row Y n,: = yn T, Φ is a K L atrix with eleents Φ k,l = φ l (x k ), R is a K N atrix with eleents R (k, n ) = r,k n corresponding to the responsibility atrix for a fixed cluster, and G is a K K diagonal atrix with eleents G (k, k ) = N n= rn,k. This approxiation is no longer guaranteed to increase the likelihood, but it yields good results in practice. Log Evidence Approxiation: Incorporating the Bayesian prior over the weights (), the arginal data likelihood is coputed by integrating over the weight atrix: p({y n } Θ ) = p({y n } Θ)p(W {λ }). (5) Where Θ = {θ } M = is the set of paraeters without {W }. This integration is intractable, however, we eploy a quadratic approxiation as suggested by (MacKay 99). The idea is to approxiate the full posterior using a Gaussian approxiation about soe ode, W. We define the function: S(θ, w) = log{p({y n } θ )p(w λ )}. () This can be used to approxiate (5) as: p({y n } θ ) = e { S(θ,w)} dw e { S(θ,w )} e { (w w )T H (w w )} dw = e { S(θ,w )} (π) MD/ H (7) using a second order Taylor expansion of the log of the integrand. The Hessian atrix H is given by the second derivative of S with respect to w. The derivative involves coputing ters that are quadratic in w which akes the coputation expensive. For this reason, we eploy a GTM approxiation once ore as described in (Bishop, Svensen, and Willias 998a). The resulting H atrix is block diagonal with each block of the for β Φ T G Φ. Now the log evidence, log p({y n } θ ) for each can be coputed as: L D (θ ) W λ w H + MD log λ. (8) Updating λ and β: Next, we use (8) to update the paraeters λ and β by finding fixed points of the corresponding derivatives. Assuing soe constant w, the updates are given by: β { = ν ND γ λ = γ w, (9) r n,k D Q (D α Q ) dn,,k (α )D d n,,k α (D α )Q }, () where d n,,k = qn,k, d n,,k = qn T,k W Ψ k W T q n,k. The paraeter γ can be interpreted as the effective nuber of paraeters and is given by: γ = MD i= ζ i α ζ i, () where ζ i are the eigenvalues of H. Updating α: The derivative of the log likelihood with respect to α leads to a cubic function in α given by: with coefficients: where v, = α x + α x + α x + x =, () x = ν NQ D, x = ν NQ D(D + Q ) β (D Q )v, β DQ v,, x = ν NQ D + β DQ v,, x = β Dv,, r,kd n n,,k and v, = r,kd n n,,k. Ideally, the likelihood should be evaluated for each possible solution of the cubic polynoial. However, this is expensive. In practice, we pick the solution α that is closest. By autoatically updating α, we are able to avoid a potentially ore expensive cross validation as used by (Chang and Ghosh ).

5 The E-Step and M-step are alternated until the expected log-likelihood or the paraeter estiates do not change significantly over consecutive iterations. At convergence, the data anifolds are given by the learned paraeters Θ and the soft clustering is encoded in the resulting responsibility (8). A hard clustering can be coputed by selecting the ost likely cluster: = arg ax {...M} r,k n. () One advantage of the EM algorith is that it can be trained increentally by feeding data saples in batches or one at a tie (Depster, Laird, and Rubin 977). This is useful when dealing with large data-sets. Initialization and paraeter selection The EM algorith is guaranteed to locally axiize the log-likelihood, however, there is no guarantee of finding a global axiu. For this reason, initialization of the odel paraeters is critical to the success of the algorith. In our experients, we used our prior assuptions to bias the optiization close to a good optiu. We also noticed that the paraeters {α, λ } that were autoatically updated were less sensitive to initialization. Thus we were able to avoid ore expensive cross validation. First we found a hard clustering of the data using our prior assuptions about the for of the final clusters. For instance, when the clusters were expected to be approxiately convex, we found that k-eans found a good initial cluster estiate. For ore coplicated clusters, we initialized using hierarchical clustering. Other initial clustering ethods that satisfy prior assuptions can also be used. The probability of each cluster, ν was then initialized as the quotient of the nuber of points in cluster and N. The anifold estiates were initialized based on the aount of non-linearity assued in each generating anifold. If the generating anifold was expected to have a low curvature, we initialized our estiate of the anifold to approxiate PCA in each cluster by solving a siple least squares proble (Bishop, Svensen, and Willias 998b). When the anifold was assued to have a high curvature, each cluster was ebedded using soe non-linear diensionality reduction algorith; in our case, ISOMAP. The weights were then initialized approxiate the apping fro the ebedded points to the data space. The noise inverse variance β was initialized as the iniu of the inverse variance lost in the initial clustering and average spacing of the projected grid. For PCA, the variance lost can be approxiated by the Q + eigenvalue of the data covariance atrix. The paraeter α was initialized to and λ was initialized to. though in practice, we found that the initial values were not critical. The latent space was selected as a Q diensional hypercube [, ] Q. However, when the anifold was expected to be closed, we odelled the latent space as a unit hypersphere S Q R Q+. K latent points were generated by sapling uniforly over this latent space. The nuber of latent points is only liited by coputational constraints. A large K iproves the odeling perforance. On the other k hand, if the latent space is insufficiently sapled, the PPS centers in the data space will lie too far away fro one another and the apping loses the topological constraints. The centers of the RBF were sapled on a grid over X. We initialized both K and L to.5n, suitably rounded. The length scale of the RBF; σ was the only cross validation paraeter we used in our experients. This paraeter was set to a sall ultiple of the distance between the RBF eans. MiPPS assues that the diensions of all the local clusters Q are known. We used the estiate of Q learned by the axiu likelihood ethod described in (Levina and Bickel 5) which we found agreed with our prior intuition about the diensionality of the coponent anifolds.. Local and Global Visualization A functional odel for the coponent anifolds can be found directly using the learned RBF apping fro a continuous latent space: W φ(x) x X. An alternative is to odel the coponent anifolds using linear interpolation between the apped latent points {W φ(x k )} K k=. If desired, splines or other non-linear interpolation techniques can also be used. If the latent space is one or two diensional, the data set can be visualized by projections to each local anifold using the posterior distribution: p(x k y n, ) = The ean of the projected point is: p(x k )p(y n, x k ) K k = p(x k )p(y n, x k ). () K x y n, = x k p(x k y n, ). (5) k= The ode can also be used if the posterior is expected to be ulti-odal. This projection ethod gives local visualizations. The PPS can be interpreted as a constrained Mixture of Factor Analyzers (MFA) odel (Tipping and Bishop 999) where each of the MFA eans are constrained to lie on soe non-linear anifold, and the noise covariances are given by gradients and globally shared paraeters. This is in contrast to the general MFA where each local subspace is coputed to approxiate the local principal coponents of the data. The effect of this constraint is that the local subspaces are coputed using the apping function as E // (x) avoiding several singular vector coputations. The noise variance paraeters are also shared along the entire anifold. This can be a useful property when the data is high diensional and sparse as we will discuss in Section 5.. The use of gradients to define the subspaces also naturally aligns the subspaces avoiding the degeneracies addressed by (Verbeek, Vlassis, and Kröse ). We will exploit this MFA interpretation to define a global ebedding. Teh and Roweis developed an alignent algorith for anifolds described using several local linear projections called the local linear co-ordination (LLC) algorith (Teh and Roweis ). First the odel learns a ixture of factor analyzers to describe the data anifold. Next the algorith

6 finds a global ebedding of these local representations using only linear transforations of the local projections and the coputed responsibilities. By interpreting the learned MiPPS odel as a MFA, the LLC algorith can be applied directly to find a global ebedding for visualization. MiPPS local co-ordinates are given by the projection of the data to each local subspace. z, = E // (x k )y n and the responsibilities are given by (8). We oit the details of the algorith ipleentation due to space constraints. We also note that LLC does not require that the local projections z, be two diensional in order to find a global twodiensional ebedding. Intuitively, MiPPS and LLC can be used to learn an unsupervised ebedding of the dataset guided by the clustering. This leads to a visualization that respects the topology of the data, but is also biased to preserve the learned clusters Results We now present results applying MiPPS to three toy data sets. In all the synthetic data experients, the anifolds are sapled in Gaussian noise and are represented by linearly interpolating the MiPPS eans. MiPPS was used to learn the partitioned S-curve randoly sapled in high noise (Fig. ). The algorith was initialized using k-eans to find two clusters. The weights were initialized to approxiate PCA on each cluster. The noisy double helix (Fig. ) Figure : Clustering the double helix. the weights were initialized to approxiate each of the two principal coponents of the entire data set Figure : Clustering two intersecting spirals. Figure : Clustering the partitioned S-curve. was ore challenging because of the intertwined anifolds. Initialization using k-eans led to the top and botto halves being separated and learned as the coponent anifolds and MiPPS was not able to escape this local axiu. Instead, we used hierarchical clustering to learn a ore representative partition. The weights were initialized to approxiate the ISOMAP ebedding. We also attepted to learn two intersecting anifolds for the intersecting spirals data set (Souvenir ). Here, the prior inforation given was that the anifolds intersect. An initial clustering step could be used to estiate the intersecting clusters. However, we found that the clustering step was not necessary for good results. The odel shown in Fig. was learned when Hyperspectral Data Hyperspectral data consists of sensor easureents by satellites or easureent aircraft easured over various locations in soe region. Each feature vector consists of hundreds of frequency bands providing fine spectral inforation about the easured area. In our experients, we used Hyperion data acquired by the NASA EO- satellite over the Okavango Delta, Botswana in May. Each data point consists of D = 5 bands. The land cover is classified into 9 types by doain experts. We selected three of the ost populous classes as shown in Table. The total nuber of data saples was N = 9 so this dataset is high diensional but sparse. The easured classes are known to vary over location, so intuitively, there

7 Class Nuber of saples Priary floodplain 7 Riparian 8 Firescar MiPPS GTM MFA Table : Class Descriptions. should be two degrees of freedo in the data corresponding to the location of the easureent. The axiu likelihood diension estiate (Levina and Bickel 5) of the data was.589 atching our intuition. Correspondingly, we set Q = for each cluster. MiPPS was initialized using the clusters found by a ixture of PPCA. The weights were learned to approxiate PCA in each cluster. We separated the data randoly into training/test pairs with the percentage of training data between % and %. We recorded the perforance of each odel on the test set over iterations using test error and average negative conditional log likelihood (ANCLL) as perforance etrics. ANCLL is coputed as N log p(ĉ yn ) where ĉ is the true class of y n. The ANCLL is a good etric for probabilistic algoriths as it gives a sense of how far the odel strays fro the correct cluster selection. For % training data, we report the results fro the training data only. For both etrics, lower values indicate better perforance. The RBF length scale paraeter was selected by cross validation using values over a logarithic scale fro.5 to.8 ultiplied by the average distance between the latent nodes. We report the results for the length scale with the lowest training error averaged over all iterations. We copared the perforance of MiPPS to related unsupervised algoriths. For co-located easureents, the ultivariate Gaussian distribution is often used by doain experts as a good class dependent feature odel for hyperspectral data (Manolakis et al. ). However, as the easureent locations are varied, the Gaussian odel becoes less accurate. This otivated the use of a Gaussian ixture odel as the base odel for unsupervised clustering of the data. Unfortunately, we were unable to learn a Gaussian ixture odel as the data is sparse. Instead, we used a ixture of two diensional factor analyzers. We also copared the algorith perforance to a ixture of GTM initialized identically to MiPPS. This was achieved by fixing each α =. We attepted to cluster the dataset using k-anifolds algorith. Unfortunately, the training process failed to converge due to widely separated cluster anifolds. This suggests that the clusters did not clearly intersect as assued by k-anifolds. All three algoriths perfored well in clustering the data set using test error and ANCLL as the perforance etrics. The results reported are the average perforance over iterations. As shown in Fig. and Fig. 5, MiPPS outperfored the MFA and ixture of GTM algorith. It was also clear that the ixture of GTM algorith gave a high probability to the correct clusters while the MFA odel was less certain about the selections. Surprisingly, the MFA outperfored the ixture of GTM in ters of test error until about % of the data was used for training. Average Test Error % of Training data Average ANCLL 5 MiPPS GTM MFA Figure : Average Test error % of Training data Figure 5: Average ANCLL Finally, we applied the LLC algorith to find a global ebedding. LLC requires a nearest neighbor paraeter which we set as though we found that the paraeter did not have uch effect on the global ebedding learned. Fig. shows one ebedding learned for the dataset. The ebedding shows good separation by class. The odel learned had a likelihood of., a test error of. and an ANCLL of.8 with α = [..5.7] suggesting that uch of the noise was orthogonal to the estiated generating anifold. In contrast, the LLC algorith initialized by MFA learned the ebedding shown in Fig. 7. The LLC visualization was unable to separate the Priary Floodplain and Firescar classes. We were only able to fit up to MFA s to the data due to sparsity. This highlights another advantage of the MiPPS odel cobined with LLC for data visualization as it can facilitate learning a detailed odel even when the data is too sparse for less constrained odels.

8 5 Priary Floodplain Riparian Firescar 7 5 Priary Floodplain Riparian Firescar Figure : Global ebedding learned by MiPPS and LLC. Figure 7: Global ebedding learned by MFA and LLC.. Conclusion In this paper, we discussed the ulti-anifold clustering proble. We described MiPPS: a principled probabilistic technique for anifold clustering. We derived the data likelihood and showed that this odel can be trained using an EM algorith. We showed the use of a Bayesian prior and showed that the resulting odel can be optiized using siple approxiations. We also showed how one can bias the algorith towards a good optiu by initializing based on prior intuition on the for of the data clusters while iniizing the aount of cross validation required to choose paraeters. We explored the versatility of the odel to show that we can odel a large class of anifolds in a principled anner. The use of a probabilistic odel allowed us to learn in uncertainty such as when the coponent anifolds intersected or were sapled in high noise. We also explicitly coputed a odel for the anifold clusters which we used lo learn a global two diensional visualization of the dataset. We deonstrated this technique in separating and visualizing land cover types in hyperspectral data. Acknowledgents: This research was supported in part by NSF (Grant IIS-758). References Bishop, C. M.; Svensen, M.; and Willias, C. K. I. 998a. Developents of the generative topographic apping. Neurocoputing :. Bishop, C. M.; Svensen, M.; and Willias, C. K. I. 998b. GTM: The generative topographic apping. Neural Coputation :5. Chang, K., and Ghosh, J.. A unified odel for probabilistic principal surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence :. Depster, A. P.; Laird, N. M.; and Rubin, D. B Maxiu likelihood fro incoplete data via the EM al- gorith. Journal of the Royal Statistical Society B(9): 8. Hadid, A., and Pietikinen, M.. Efficient locally linear ebeddings of iperfect anifolds. In Machine Learning and Data Mining in Pattern Recognition, volue 7 of Lecture Notes in Coputer Science. Hastie, T. 98. Principal curves and Surfaces. Ph.D. Dissertation, Departent of Statistics, Stanford University. Levina, E., and Bickel, P. J. 5. Maxiu likelihood estiation of intrinsic diension. In NIPS 7. Cabridge, MA: MIT Press MacKay, D. J. 99. Bayesian interpolation. Neural Coputation :5 7. Manolakis, D. G.; Marden, D.; Kerekes, J. P.; and Shaw, G. A.. Statistics of hyperspectral iaging data. In Society of Photo-Optical Instruentation Engineers (SPIE) Conference Series, volue 8, 8. Souvenir, R. M.. Manifold learning for natural iage sets. Ph.D. Dissertation, Departent of Coputer Science, Washington University. Teh, Y. W., and Roweis, S.. Autoatic alignent of local representations. NIPS 5:8 88. Tino, P., and Nabney, I.. Hierarchical GTM: Constructing localized nonlinear projection anifolds in a principled way. IEEE Transactions on Pattern Analysis Machine Intelligence (5):9 5. Tipping, M. E., and Bishop, C. M Mixtures of probabilistic principal coponent analyzers. Neural Coputation : 8. Verbeek, J.; Vlassis, N.; and Kröse, B.. Coordinating principal coponent analyzers. In Proceedings of the International Conference on Artificial Neural Networks. Yankov, D., and Keogh, E. J.. Manifold clustering of shapes. In Proceedings of the International Conference on Data Mining. 5