Experimental Evaluation of Latent Variable Models. for Dimensionality Reduction


 Kerry Anthony
 25 days ago
 Views:
Transcription
1 Experimental Evaluation of Latent Variable Models for Dimensionality Reduction Miguel Á. CarreiraPerpiñán and Steve Renals a Dept. of Computer Science, University of Sheffield th IEEE Signal Processing Society Workshop on Neural Networks for Signal Processing (NNSP8) Aug. Sep., 8, Cambridge, UK a This work has been supported by a scholarship from the Spanish Ministry of Education and Science, by a ESPRIT Long Term Research Project SPRACH (77) and by an award from the Nuffield Foundation.
2 Electropalatography (EPG) A plastic pseudopalate fitted to a person s mouth detects the presence or absence of contact between the tongue and the palate in 6 different locations during an utterance (sampled at Hz). Result: sequence of 6dimensional binary EPG frames. Data reduction necessary, traditionally via fixed linear indices. ACCORII database: synchronised data (EPG, acoustic, etc.) for different utterances and speakers. The mapping phonemetoepg is not onetoone, e.g. / or /. Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
3 Electropalatography (cont.) wires to PC lips palate teeth teeth wires to PC teeth teeth lips velum palate electrodes velum electrodes Pseudopalate and representative EPGs for the typical stable phase of different phonemes. Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
4 NNSP8, AUG. SEP., 8, CARIDGE, UK  The Reading pseudopalate. Sfrag replacements EXPERIMENTAL EVALUATION OF LATENT VARIABLE MODELS FOR DIMENSIONALITY REDUCTION
5 Latent variable models Prior p(x) Induced p(t Θ) t x x f f(x; Θ) t Manifold M t t x Latent space of dimension L = Data space of dimension D = Marginalisation in latent space: p(t) = p(t x)p(x) dx. Maximum likelihood parameter estimation: l(θ) = N n= log p(t n Θ). Inverse mapping given by informative point (mean, mode) of posterior: p(x t) = p(t x)p(x) p(t). Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
6 Examples of latent variable models Factor analysis: prior is normal N (, I), mapping is linear, noise model is normal with diagonal covariance matrix. : like but noise model has isotropic covariance. : prior is uniform over discrete latent grid, mapping is a generalised linear model, noise model is normal with isotropic covariance matrix. Mixtures of factor analysers (one mean parameter and one factor per analyser, noise model covariance matrix common to all analysers). We also tried mixtures of multivariate Bernoulli distributions (not really a latent variable model). All these models can be trained via an EM algorithm. Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
7 Factors / prototypes (speaker RK) λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ M Λ µ π =. Λ µ π =. Λ µ π =. Λ µ π =.7 p π =.5 p π =. p π =.7 p π =. p 5 π 5 =.5 p 6 π 6 =. p 7 π 7 =. p 8 π 8 =.5 p π =. Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction 5
8 and reconstruction error (speaker RK) Training set Test set x 5 5 x M M Squared reconstruction error M M Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction 6
9 ! " #,. # ) *,+  %, Twodimensional representation (speaker RK) Factor analysis frag replacements Factor Factor Factor Factor Trajectory in latent space of the highlighted utterance fragment I prefer Kant to Hobbes for a good bedtime book ( #(! " $&% /. ). Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction 7
10 Conclusions Adaptive methods outperform fixed data reduction indices. and performed similarly in terms of likelihood. Mixtures of factor analysers and multivariate Bernoulli distributions did not perform well. Twodimensional outperformed all other methods in terms of likelihood and error reconstruction and reveals nonlinear structure in the data. This suggests a low intrinsic dimensionality for the EPG data. Additional results available via the web at miguel/research/epg.html Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction 8
11 Factors / prototypes (speaker HD) λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ M Λ µ π =. Λ µ π =.6 Λ µ π =. Λ µ π =.6 p π =. p π =. p π =. p π =. p 5 π 5 =. p 6 π 6 =.7 p 7 π 7 =. p 8 π 8 =.7 p π =. Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
12 and reconstruction error (speaker HD) Training set Test set Squared reconstruction error x M M 6 6 x M M Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
13 ! " #,. # ) *,+  %, Twodimensional representation (speaker HD) Factor analysis 7 frag replacements Factor Factor Factor Factor Trajectory in latent space of the highlighted utterance fragment I prefer Kant to Hobbes for a good bedtime book ( #(! " $&% /. ). Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
14 !!! # ) *,+  %,!! " #,. Discontinuities in latent space (speaker RK) Factor analysis.5.8 Factor Frame 5 Frame Factor Frame 7 Selected subsequence of the utterance fragment I prefer Kant to Hobbes for a good bedtime book ( #! " $ % /. ). The abrupt transition from to (frames 5 6) produces a discontinuity in latent space. Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
15 factors before and after varimax rotation (speaker HD) BEFORE λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ AFTER λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ Both sets of components span the same linear subspace, but the varimaxrotated one is more easily interpretable. Miguel Á. CarreiraPerpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction