Extracting Coactivated Features from Multiple Data Sets

Transcription

1 Extracting Coactivated Features from Multiple Data Sets Michael U. Gutmann University of Helsinki Aapo Hyvärinen University of Helsinki Michael U. Gutmann University of Helsinki ICANN 11 - p. 1

2 Contents Introduction Extraction of coactivated features This talk is about a new method to find related features (structure) in multiple data sets. Testing on artificial data Application to real data Summary Background information on the extraction of related features from multiple data sets Explanation of the statistical model underlying our method Testing our method on artificial data Application to real data (here: natural images, in the paper: also brain imaging data) Michael U. Gutmann University of Helsinki ICANN 11 - p.

3 Introduction Michael U. Gutmann University of Helsinki ICANN 11 - p. 3

4 An example 15 Data set 1 Data set Goals: 1. Characterize each data set separately Eigenvalues and eigenvectors of covariance matrices. Find relations between the two data sets Michael U. Gutmann University of Helsinki ICANN 11 - p.

5 Correlation based method (1/3) Whitened data set 1 (normalized representation) After change of basis: data is still white Whitening is defined up to a rotation Choose coordinate systems which best describe the relation between the two data sets 1 and Michael U. Gutmann University of Helsinki ICANN 11 - p. 5

6 Correlation based method (/3) 3 1 Whitened data set 1 Whitened data set 3 e e 1 α 1 1 α Correlation between projections one i α α.5 The x-coordinate of a data point in the coordinate system defined by e i is given by its projection on e i. Compute the correlation between the x-coordinates for different coordinate systems. Choose the coordinate systems for which the x-coordinates are most strongly correlated (here: correlation coefficient of.). Described method is called Canonical Correlation Analysis (CCA). Michael U. Gutmann University of Helsinki ICANN 11 - p.

7 Correlation based method (3/3) Data set Whitened data set 1 Data set Whitened data set Correlation between projections on ei e1 α1 e α α α Method does not seem to work here. Michael U. Gutmann University of Helsinki ICANN 11 - p. 7

8 What does not work mean? It means 1. that we did not find meaningful features within each data set. that we did not find any relation between the two data sets Whitened data set 1 Axis 1 1 Projection on axis Blue: original data Red: data with same marginal distr. Whitened data set Axis Projection on axis 1 Identify independent components Michael U. Gutmann University of Helsinki Identify coactivation ICANN 11 - p. 8

9 In this talk... Introduction CCA Limitations New method Extraction of coactivated features Testing on artificial data Application to real data Summary I will present a method where 1. the features for each data set are maximally statistically independent. the features across the data sets tend to be jointly activated: they have statistically dependent variances 3. multiple data sets can be analyzed Michael U. Gutmann University of Helsinki ICANN 11 - p. 9

10 Extraction of coactivated features Michael U. Gutmann University of Helsinki ICANN 11 - p. 1

11 Statistical model underlying our method (1/) Introduction Extraction of coactivated features Given n data sets, we assume that each set is formed by iid. observations of a random vector zi Rd. To model structure within a data set, we assume that Statistical model Learning zi = Testing on artificial data d X qik sik (i = 1,... n) k=1 Application to real data qik : orthonormal, si1,..., sid : statistically independent To model structure across the data sets, we assume that s1k,..., snk are statistically dependent. q11 q1 q1 Michael U. Gutmann University of Helsinki 5 q 1 z z1 Blue: original data Red: data with same marginal distr. s1 Summary z11 z s ICANN 11 - p. 11

12 Statistical model underlying our method (/) Introduction Extraction of coactivated features Statistical model Learning Testing on artificial data Application to real data Summary Dependency assumptions: The k-th sources from all the data sets share a common (latent) variance variable σ k : s 1 k = σ k s 1 k s k = σ k s k... s n k = σ k s n k The s 1 k,..., sn k correlated). are Gaussian (zero mean, possibly Choosing a prior for σ k completes the model specification. Linear correlation 1 Correlation of squares 1 Blue: original data Red: data with same marginal distr. 1 s s s 1 s 1... s s 1 1 s 1 s 1 s s 1 1 s 1 s 1 s 1 1 s 1 Michael U. Gutmann University of Helsinki ICANN 11 - p. 1

13 Applying the method learning the parameters Introduction Extraction of coactivated features Statistical model Learning Testing on artificial data Application to real data Summary The most interesting parameters of the model are the features q i k. They can be learned by maximizing the log-likelihood l(q 1 1,...,q n d ). For the special case of uncorrelated sources, ( T d n ) l(q 1 1,...,q n d) = G k (q i k T z i (t)) t=1 k=1 i=1, (1) where pσk (σ k ) G k (u) = log (πσk )n exp ( u σk ) dσ k. () Equations show that the presented method is related to Independent Subspace Analysis (see paper) Michael U. Gutmann University of Helsinki ICANN 11 - p. 13

14 Testing on artificial data Michael U. Gutmann University of Helsinki ICANN 11 - p. 1

15 Simulation setup and goals Introduction Extraction of coactivated features Testing on artificial data Setup Results Application to real data Summary Setup Three data sets (n = 3) of dimension four (d = ) No linear correlation in the sources s i k Randomly chosen orthonormal mixing matrices Q 1,Q,Q 3 1 observations Learning of the parameters by maximization of log-likelihood, with ad hoc nonlinearity G(u) =.1+u Quantities of interest Error in the mixing matrices Identification of the coupling Michael U. Gutmann University of Helsinki ICANN 11 - p. 15

16 Results. 3 x 1 3 Introduction Extraction of coactivated features Testing on artificial data Setup Results Application to real data Objective Squared estimation error Summary Correct coupling (fraction) Correct coupling (fraction) Results for one estimation problem. Optimization performed for different initializations. Correct coupling at maximum of the log-likelihood Presence of local maxima (not nice! but no catastrophe, see following slides) Learning the right mixing matrices without learning the right coupling seems possible. Michael U. Gutmann University of Helsinki ICANN 11 - p. 1

17 Application to real data Michael U. Gutmann University of Helsinki ICANN 11 - p. 17

18 Setup Introduction Extraction of coactivated features Testing on artificial data Application to real data Setup Results Summary Data set 1: 1 image patches of size 5px 5px, extracted at random locations from natural video data Data set : same image patches, ms later For each data set separately: whitening and dimension reduction (98% of variance retained) Learning of 5 features per data set by maximization of the log-likelihood of the model (same objective function as for the artificial data). Michael U. Gutmann University of Helsinki ICANN 11 - p. 18

19 Learned features Introduction Our method: First data set Second data set Extraction of coactivated features Testing on artificial data Application to real data Setup Results Summary Canonical correlation analysis: First data set Second data set Michael U. Gutmann University of Helsinki ICANN 11 - p. 19

20 Summary Michael U. Gutmann University of Helsinki ICANN 11 - p.

21 Summary Introduction Extraction of coactivated features Testing on artificial data Application to real data Summary Summary Presented a new method to find related features (structure) in multiple data sets: 1. The features for each data set are maximally statistically independent.. The features across the data sets tend to be jointly activated: they have statistically dependent variances. 3. Multiple data sets can be analyzed. In the paper: more theory (in particular, more on the relation to CCA) more simulations with natural images simulations with brain imaging data Michael U. Gutmann University of Helsinki ICANN 11 - p. 1