Regularized Tensor Factorizations & Higher-Order Principal Components Analysis

Transcription

1 Regularized Tensor Factorizations & Higher-Order Principal Components Analysis Genevera I. Allen Department of Statistics, Rice University, Department of Pediatrics-Neurology, Baylor College of Medicine, & Jan and Dan Duncan Neurological Research Institute, Texas Children s Hospital. August 2, 2012 G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

2 1 Introduction Motivation Background 2 Sparse HOPCA Algorithmic Methods: Sparse HOSVD, HOOI & CP-ALS Deflation Approaches to Sparse HOPCA 3 Results Simulations Example: StarPlus fmri Data G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

3 High-Dimensional Multi-way Data Examples: Neuroimaging. Microscopy. Hyperspectral Imaging. Remote Sensing. Bibliometrics. Chemometrics. Environmetrics. Network Data. Internet Data. Hyperspectral Image of Gulf Oil Spill 2010 (SpecTIR). G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

4 Principal Component Analysis PCA for Matrix Data: Exploratory Data Analysis. Pattern Recognition. Dimension Reduction. Data Visualization. For High-Dimensional data, performance can be improved by regularization... Sparse PCA. (Jollieffe et al., 2003; Johnstone and Lu, 2004; Zou et al, 2006; Shen and Huang, 2008) Functional PCA. (Silverman, 1996; Ramsay, 2006) Two-way Regularized PCA. (Huang et al., 2009, Witten et al., 2010, Lee et al., 2011, Allen et al., 2011) G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

5 Functional MRIs StarPlus Data (Subject 04847): (Mitchell et al., 2004) 20 tasks in which sentence agrees with image. 20 tasks in which sentence opposes image. Each task lasted 27 seconds (55 time points). Images: Data Set: 4,698 voxels 40 tasks 55 time points. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

6 Functional MRIs Classical PCA on Flattened Tensor: G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

7 Functional MRIs Sparse Generalized PCA on Flattened Tensor: G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

8 1 Introduction Motivation Background 2 Sparse HOPCA Algorithmic Methods: Sparse HOSVD, HOOI & CP-ALS Deflation Approaches to Sparse HOPCA 3 Results Simulations Example: StarPlus fmri Data G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

9 Tensor Decompositions and Higher-Order PCA Review PCA for matrix X: maximize v k v T k XT X v k subject to v T k v k = 1, & v T k v j = 0 j < k. Or... minimize U,V,d 1 2 X U D VT 2 F subject to U T U = I, V T V = I, & d 0. Solved via the Singular Value Decomposition (SVD). How do we perform PCA without flattening a tensor (Higher-Order PCA)? No equivalent of the SVD in 3 or more dimensions! G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

10 Tensor Decompositions and Higher-Order PCA Tensor Notation: X R n p q X a tensor, X a matrix, x a vector, x a scalar. Flattened (or matricized) tensor: X (1) R n pq. Tensor Frobenius norm: X F = X 2 ijl. Tensor mode multiplication: if U R n K, X 1 U R K p q. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

11 Tensor Decompositions and Higher-Order PCA CANDECOMP / PARAFAC (CP) Decomposition: CP values: d k 0. X K k=1 d k u k v k w k CP factors: u k R n 1 : u T k u k = 1, v k R p 1 : v T k v k = 1, and w k R q 1 : w T k w k = 1. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

12 Tensor Decompositions and Higher-Order PCA Tucker Decomposition: Tucker core: D R K K K. X D 1 U 2 V 3 W Tucker factors: U R n K : U T U = I, V R p K : V T V = I, and W R q K : W T W = I. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

13 Objectives Why Regularization in Higher-Order PCA? Feature selection, smoothing, data compression, data visualization,... Existing Work..... Non-Negative Tensor Factorizations: Cichocki et al., 2009; Hazan et al., 2005; Morup et al., 2008; Lim and Comon, 2009; Liu et al., 2011; Chi and Kolda, Sparse Non-Negative Tensor Factorizations: Ruiters and Klien, 2009; Pang et al., 2011; Cichocki et al., Limited Attention from the Statistics Community. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

14 Objectives Goal Develop a mathematically sound, computationally attractive, and flexible method to incorporate regularization in each tensor factor, or HOPC. In this talk we... 1 Formulate two algorithmic methods for Sparse HOPCA: Sparse Tucker-ALS & Sparse CP-ALS. 2 Develop deflation-based Tensor Power Algorithm and related Sparse CP-TPA method. 3 Propose extensions to Generalized HOPCA and Functional HOPCA & Non-negative HOPCA. 4 Present a case study on fmri data. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

15 1 Introduction Motivation Background 2 Sparse HOPCA Algorithmic Methods: Sparse HOSVD, HOOI & CP-ALS Deflation Approaches to Sparse HOPCA 3 Results Simulations Example: StarPlus fmri Data G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

16 Sparse HOOI Higher-Order Orthogonal Iteration (HOOI) or Tucker-ALS: 1 Repeat until convergence: 1 U First K principal components of (X 2 V 3 W) (1). 2 V First K principal components of (X 1 U 3 W) (2). 3 W First K principal components of (X 1 U 2 V) (3). 2 D X 1 U 2 V 3 W. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

17 Sparse HOOI Sparse Higher-Order Orthogonal Iteration (HOOI): 1 Repeat until convergence: 1 U First K sparse principal components of (X 2 V 3 W) (1). 2 V First K sparse principal components of (X 1 U 3 W) (2). 3 W First K sparse principal components of (X 1 U 2 V) (3). 2 D X 1 U 2 V 3 W. Questions: What optimization problem is this solving? Can we guarantee convergence? Is this efficient computationally? G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

18 Sparse CP-ALS CP Alternating Least Squares (CP-ALS): 1 K 2 X d k u k v k w k 2 F k=1 1 2 X (1) U(V W) T 2 F Algorithm: 1 Solve for each factor via simple least squares with the others fixed. 2 Normalize the factor so the columns are norm one. 3 Repeat until convergence. (Khatri-Rao Product: V W = [v 1 w 1... v K w K ] R pq K.) G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

19 Sparse CP-ALS Replace Least Squares with LASSO (l 1 -norm penalty): minimize U 1 2 X (1) U(V W) T 2 F + λ u U 1 Employ same algorithm structure as CP-ALS. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

20 Sparse CP-ALS What optimization problem is this algorithm solving? minimize U,V,W,d 1 K 2 X d k u k v k w k 2 F k=1 + λ u U 1 + λ v V 1 + λ w W 1 NOT TRUE! G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

21 1 Introduction Motivation Background 2 Sparse HOPCA Algorithmic Methods: Sparse HOSVD, HOOI & CP-ALS Deflation Approaches to Sparse HOPCA 3 Results Simulations Example: StarPlus fmri Data G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

22 Tensor Power Algorithm Rank-one CP Optimization Problem: minimize u,v,w,d X d u v w 2 2 subject to u T u = 1, v T v = 1, w T w = 1, & d > 0. maximize u,v,w X 1 u 2 v 3 w subject to u T u = 1, v T v = 1, & w T w = 1. Can update one factor at a time with the others fixed. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

23 Tensor Power Algorithm Algorithm (Power Method or Deflation Approach): 1 For k = 1... K 1 Repeat until converge: 1 u k ˆX 2 v k 3 w k ˆX 2 v k 3 w k 2. 2 v k ˆX 1 u k 3 w k ˆX 1 u k 3 w k 2. 3 w k ˆX 1 u k 2 v k ˆX 1 u k 2 v k 2. 2 d k ˆX 1 u k 2 v k 3 w k. 3 ˆX ˆX d k u k v k w k. Greedy! Computes rank-one solution on residuals of previous factorization. Converges to a local optimum of the rank-one CP problem. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

24 Sparse CP-TPA: Optimization Problem Rank-one Sparse CP-TPA Problem: maximize u,v,w X 1 u 2 v 3 w λ u u 1 λ v v 1 λ w w 1 subject to u T u 1, v T v 1, & w T w 1. where λ u, λ v, λ w 0 are regularization parameters controlling the amount of sparsity. Tri-concave relaxation of rank-one CP optimization problem. Concave in u with v and w fixed and vice versa. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

25 Sparse CP-TPA: Solution & Algorithm Solution û = S(X 2 v 3 w, λ u ). ˆv = S(X 1 u 3 w, λ v ). ŵ = S(X 1 u 2 v, λ w ). where S(, λ) = sign( )( λ) + is the soft-thresholding operator. Then, the coordinate-wise solutions to the Sparse CP-TPA problem are: { { { û u û = 2 û 2 > 0 ˆv 0 otherwise, v ˆv = 2 ˆv 2 > 0 ŵ 0 otherwise, & w ŵ = 2 ŵ 2 > 0 0 otherwise, which when repeated monotonically increase the objective and converge to a local maximum. Multiple Factors: Tensor Power Algorithm deflation approach. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

26 Advantages of Sparse CP-TPA 1 Convergence to a local optimum assured. 2 Monotonic algorithm Track progress. 3 Scale of the factors directly controlled Numerically stable algorithm and solution. Solution for each factor either has norm one or is exactly zero. 4 Solution for each factor a simple analytical form Iterative updates computationally inexpensive. 5 Require less computer memory as only need to compute components of final solution. 6 General framework yields many fruitful extensions. Depends heavily on initial starting point... as do all HOPCA and Sparse HOPCA methods. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

27 Sparse HOPCA: Loose Ends Selecting regularization parameters: Cross-validation. BIC: Each sub-problem solves a penalized regression problem. Stability Selection. Amount of Variance Explained: Projection Matrices: P (U) k = U k (U T k U k) 1 U T k analogously. ) and P(V k, P (w) k Cumulative Proportion of Variance Explained by first k HOPC s: X 1 P (U) k 2 P (V ) k 3 P (W ) k X 2 F 2 F. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

28 Extensions Flexible Modern Multivariate Methods (can be used in combination): 1 General Penalties (Allen et al., 2011): Penalties that are norms or semi-norms. Re-scaled solution of a penalized regression problem. Examples: group norms, total variation norms, l q norms. 2 Non-negativity (Allen & Maletic-Savatic, 2011): Soft-thresholding operator replaced by positive thresholding operator. 3 Multi-way Functional Data (Huang et al., 2009): Multi-way half-smoothing via Tucker Decomposition. Multi-way functional data penalties estimated via deflation. Two approaches are not equivalent. 4 Structured Data... Generalized HOPCA. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

29 Generalized HOPCA Allen et al. (2011) introduced Generalized PCA for heteroscedastic errors of structured data (data in which variables are associated with a fixed location). Generalized Frobenius norm with generalizing operators such as graph Laplacians or smoothing matrices based on known structure. Rank-one Generalized CP minimize u,v,w,d 1 2 X d u v w 2 Q (1),Q (2),Q (3) subject to u T Q (1) u = 1, v T Q (2) v = 1, w T Q (3) w = 1, & d > 0. Sparse Generalized CP: Extension of Sparse CP framework. Can be solved via methods introduced in Allen et al. (2011). G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

30 1 Introduction Motivation Background 2 Sparse HOPCA Algorithmic Methods: Sparse HOSVD, HOOI & CP-ALS Deflation Approaches to Sparse HOPCA 3 Results Simulations Example: StarPlus fmri Data G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

31 Overview of Simulation Results Best Sparse HOOI Best at both signal recovery & feature selection. Comparable Sparse CP-TPA Comparable in all scenarios except when all modes have equal dimension. Good Sparse HOSVD Can have erratic behavior for K > 1. Not Good Sparse CP-ALS Not optimal at feature selection. Sparse CP-TPA MUCH faster than Sparse HOOI. Full simulation results available from gallen/. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

32 1 Introduction Motivation Background 2 Sparse HOPCA Algorithmic Methods: Sparse HOSVD, HOOI & CP-ALS Deflation Approaches to Sparse HOPCA 3 Results Simulations Example: StarPlus fmri Data G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

33 Sparse GPCA on Flattened Tensor StarPlus Data: 4,698 voxels 36 tasks 55 time points. Flatten tensor to 4,698 voxels 1,098 time point tasks (only tasks with sentence - image agreement). Spatial generalizing operator: Graph Laplaican of nearest neighbor graph connecting 3D grid of voxels. Temporal generalizing operator: Kernel smoother. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

34 Sparse GPCA on Flattened Tensor G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

35 Sparse GHOPCA on Tensor Data 4,698 voxels 36 tasks 55 time points. Spatial operator: Graph Laplaican. Temporal operator: Kernel Smoother. Task operator: Identity. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

36 StarPlus Results Tucker Decomposition: G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

37 StarPlus Results Sparse GCP-TPA: G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

38 StarPlus Results Sparse GCP-TPA: G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

39 Future Work Statistics Work: 1 All methods can be trivially extended to > 3 dimensions. 2 Initializations for algorithms. 3 How many factors, K? 4 Is HOPCA and Sparse HOPCA asymptotically (in)consistent? 5 Further multivariate methods for tensors. 6 Regression & classification with tensors. Further Applications: Hyperspectral imaging, Spatio-temporal data, Multi-dimensional NMR spectroscopy, Microscopy, remote sensing, and other Neuroimaging data. G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

40 Software & Acknowledgments Software: Coming soon... Sparse HOPCA Matlab Toolbox based on the Matlab Tensor Toolbox (Bader & Kolda). Funding: National Science Foundation, Division of Mathematical Sciences G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28

41 References G. I. Allen, Regularized tensor decompositions and higher-order principal components analysis, arxiv: , Rice University Technical Report No. TR , G. I. Allen, Sparse higher-order principal components analysis, In Artificial Intelligence and Statistics, G. I. Allen, L. Grosenick & J. Taylor, A generalized least squares decomposition, arxiv: , Rice University Technical Report No. TR , G. I. Allen & M. Maletic-Savatic, Sparse non-negative generalized PCA with applications to metabolomics, Bioinformatics, 27:21, , F. D. Campbell & G. I. Allen, Algorithms and approaches for analyzing massive structured data with Sparse Generalized PCA, In Preparation, T. G. Kolda & B. W. Bader, Tensor decompositions and applications, SIAM review, 51:3, , G. I. Allen (Rice & BCM) Sparse HOPCA August 2, / 28