Experimental Evaluation of Latent Variable Models for Dimensionality Reduction Miguel Á. Carreira-Perpiñán and Steve Renals a Dept. of Computer Science, University of Sheffield {M.Carreira,S.Renals}@dcs.shef.ac.uk th IEEE Signal Processing Society Workshop on Neural Networks for Signal Processing (NNSP8) Aug. Sep., 8, Cambridge, UK a This work has been supported by a scholarship from the Spanish Ministry of Education and Science, by a ESPRIT Long Term Research Project SPRACH (77) and by an award from the Nuffield Foundation.
Electropalatography (EPG) A plastic pseudopalate fitted to a person s mouth detects the presence or absence of contact between the tongue and the palate in 6 different locations during an utterance (sampled at Hz). Result: sequence of 6-dimensional binary EPG frames. Data reduction necessary, traditionally via fixed linear indices. ACCOR-II database: synchronised data (EPG, acoustic, etc.) for different utterances and speakers. The mapping phoneme-to-epg is not one-to-one, e.g. / or /. Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
Electropalatography (cont.) wires to PC lips palate teeth teeth wires to PC teeth teeth lips velum palate electrodes velum electrodes Pseudopalate and representative EPGs for the typical stable phase of different phonemes. Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
NNSP8, AUG. SEP., 8, CARIDGE, UK - The Reading pseudopalate. Sfrag replacements EXPERIMENTAL EVALUATION OF LATENT VARIABLE MODELS FOR DIMENSIONALITY REDUCTION
Latent variable models Prior p(x) Induced p(t Θ) t x x f f(x; Θ) t Manifold M t t x Latent space of dimension L = Data space of dimension D = Marginalisation in latent space: p(t) = p(t x)p(x) dx. Maximum likelihood parameter estimation: l(θ) = N n= log p(t n Θ). Inverse mapping given by informative point (mean, mode) of posterior: p(x t) = p(t x)p(x) p(t). Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
Examples of latent variable models Factor analysis: prior is normal N (, I), mapping is linear, noise model is normal with diagonal covariance matrix. : like but noise model has isotropic covariance. : prior is uniform over discrete latent grid, mapping is a generalised linear model, noise model is normal with isotropic covariance matrix. Mixtures of factor analysers (one mean parameter and one factor per analyser, noise model covariance matrix common to all analysers). We also tried mixtures of multivariate Bernoulli distributions (not really a latent variable model). All these models can be trained via an EM algorithm. Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
Factors / prototypes (speaker RK) λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ M Λ µ π =. Λ µ π =. Λ µ π =. Λ µ π =.7 p π =.5 p π =. p π =.7 p π =. p 5 π 5 =.5 p 6 π 6 =. p 7 π 7 =. p 8 π 8 =.5 p π =. Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction 5
and reconstruction error (speaker RK) Training set Test set x 5 5 x 6 5 6 6 8 6 M 6 5 6 5 6 6 8 6 M 6 5 6 8 6 6 8 6 Squared reconstruction error 8 7 6 5 6 5 6 M 6 8 6 6 6 8 6 8 7 6 5 6 M 5 6 6 8 6 6 6 8 6 Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction 6
! " #,. # ) *,+ - %, Two-dimensional representation (speaker RK) Factor analysis 5 6 7 8.5.8 frag replacements Factor.5.5.5 Factor Factor.6.....6 8 7 6 5 8 7 5 6 5 8.5.5.5.5 Factor.8 6 7.8.6.....6.8 Trajectory in latent space of the highlighted utterance fragment I prefer Kant to Hobbes for a good bedtime book ( #(! " $&% /. ). Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction 7
Conclusions Adaptive methods outperform fixed data reduction indices. and performed similarly in terms of likelihood. Mixtures of factor analysers and multivariate Bernoulli distributions did not perform well. Two-dimensional outperformed all other methods in terms of likelihood and error reconstruction and reveals nonlinear structure in the data. This suggests a low intrinsic dimensionality for the EPG data. Additional results available via the web at http://www.dcs.shef.ac.uk/ miguel/research/epg.html Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction 8
Factors / prototypes (speaker HD) λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ M Λ µ π =. Λ µ π =.6 Λ µ π =. Λ µ π =.6 p π =. p π =. p π =. p π =. p 5 π 5 =. p 6 π 6 =.7 p 7 π 7 =. p 8 π 8 =.7 p π =. Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
and reconstruction error (speaker HD) Training set Test set Squared reconstruction error x 5 6 5 6 6 8 6 8 6 7 6 5 M 6 5 6 6 6 6 8 6 6 6 8 6 M 6 6 x 6 5 6 6 8 8 6 8 6 8 7 6 5 M 6 5 6 6 6 6 8 6 6 M 6 8 6 Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
! " #,. # ) *,+ - %, Two-dimensional representation (speaker HD) Factor analysis 7 frag replacements Factor Factor.8.6 8. 5 5 8 5 6 Factor.. 7 8 6 7 5 8 8 7 6 5 7 6. 6 5.6 5.8.5.5.5 Factor.8.6.....6.8 Trajectory in latent space of the highlighted utterance fragment I prefer Kant to Hobbes for a good bedtime book ( #(! " $&% /. ). Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
!!! # ) *,+ - %,!! " #,. Discontinuities in latent space (speaker RK) Factor analysis.5.8 Factor.5.5.5 5 6.6.....6 5 6 Frame 5 Frame 6.5.5.5.5.5 Factor 7.8.8.6.....6.8 7 Frame 7 Selected subsequence of the utterance fragment I prefer Kant to Hobbes for a good bedtime book ( #! " $ % /. ). The abrupt transition from to (frames 5 6) produces a discontinuity in latent space. Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction
factors before and after varimax rotation (speaker HD) BEFORE λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ AFTER λ λ λ λ λ 5 λ 6 λ 7 λ 8 λ Both sets of components span the same linear subspace, but the varimax-rotated one is more easily interpretable. Miguel Á. Carreira-Perpiñán and Steve Renals Experimental Evaluation of Latent Variable Models for Dimensionality Reduction