Quasi-Newton algorithm for best multilinear rank approximation of tensors

Transcription

1 Quasi-Newton algorithm for best multilinear rank of tensors and Lek-Heng Lim Department of Mathematics Linköpings Universitet 6th International Congress on Industrial and Applied Mathematics

2 Outline

3 Multidimensional arrays and multilinear tensor scalar α R vector a R I matrix A R I J tensor A R I 1 I 2 I N A R I J K zero order tensor first order tensor second order tensor N th order tensor A

4 Multidimensional arrays and multilinear tensor scalar α R zero order tensor vector a R I first order tensor matrix A R I J second order tensor tensor A R I 1 I 2 I N N th order tensor A R I J K A X S Z T Y T S R r 1 r 2 r 3 and X R I r 1, Y R J r 2, Z R K r 3

5 scalar (inner) product: A, B = i,j,k a ijkb ijk orthogonality: A, B = 0 Frobenius norm: A F = A, A multilinear matrix multiplication X, Y, Z matrices B = (X, Y, Z) A b ijk = λ,µ,ν x iλ y jµ z kν a λµν

6 scalar (inner) product: A, B = i,j,k a ijkb ijk orthogonality: A, B = 0 Frobenius norm: A F = A, A multilinear matrix multiplication X, Y, Z matrices B = (X, Y, Z) A b ijk = λ,µ,ν x iλ y jµ z kν a λµν contravariant B = A (X, Y, Z) bijk = λ,µ,ν a λµν x λi y µj z νk covariant (X T, Y T, Z T ) A = A (X, Y, Z)

7 scalar (inner) product: A, B = i,j,k a ijkb ijk orthogonality: A, B = 0 Frobenius norm: A F = A, A multilinear matrix multiplication X, Y, Z matrices B = (X, Y, Z) A b ijk = λ,µ,ν x iλ y jµ z kν a λµν contravariant B = A (X, Y, Z) bijk = λ,µ,ν a λµν x λi y µj z νk covariant (X T, Y T, Z T ) A = A (X, Y, Z) (X, Y) A = XAY T contravariant A (X, Y) = X T AY covariant A (I, I, Z) = A (Z) 3

8 and notation contracted products: A and B of appropriate sizes C = A, B 1 a λij b λkl 4-tensor c ijkl = λ C = A, B 1;2 c ijkl = λ a λij b kλl 4-tensor D = A, B 1:2 d jk = a λµj b λµk λ,µ 2-tensor D = A, B 2,3;3,1 djk = a jλµ b µkλ λ,µ 2-tensor e = A, B = A, B 1:3 e = λ,µ,ν a λµν b λµν scalar

9 , notation and properties Lemma tensor (outer) product: A R I J K, B R L M R I J K L M C = A B, useful with negative contraction subscripts c ijklm = a ijk b lm A, B 2:3 A, B 1, A, B 2 A, B (1,3). Let the N-tensors B and C and the matrix Q be of conforming dimensions. Then B (Q) i, C = Q, B, C i B (Q) i, C i = Q T B, C i = Q, B, C i 1

10 Canonical tensor matricization 3-tensors (or unfolding, or flattening) R I J K A A (1) R I JK, R I J K A A (2) R J KI R I J K A A (3) R K IJ

11 Canonical tensor matricization relation to multilinear matrix multiplication Let A be a 5-tensor and consider B = A (X, Y, Z, U, V) B (2) = Y T A (2) (X Z U V) B (3,2) = (Z Y) T A (3,2) (X U V) B (3,2;5,4,1) = (Z Y) T A (3,2;5,4,1) (V U X) The matrix cases: B = (X, Y) A = XAY T B (1) = XA (1) Y T B, B (2) = YA (2) X T B T Vectorization is possible as well B (2,1; ) = vec(b) = (Y X) vec(a) = (Y X)A (2,1; )

12 multilinear rank (Tucker model): r n = rank n (A) = rank ( A (n)) rank(a) = (r 1, r 2, r 3 ) outer product rank (Candecom/Parafac): minimal sum of rank 1 tensors (outer product of vectors)

13 Higher order SVD R I J K A = (U, V, W) C W A = U C V U, V, W orthogonal, C all orthogonal and ordered Truncated not optimal! If rank(a) = (r 1, r 2, r 3 ) then C(1 : r 1, 1 : r 2, 1 : r 3 ) 0 and the rest is zero.

14 The general problem nonlinear manifold optimization The problem min A (X, Y, Z) S A S X Z T Y T Problem overparameterized and equivalent with max (X T, Y T, Z T ) A = max A (X, Y, Z), X T X = I, Y T Y = I and Z T Z = I

15 Nonlinear optimization on a product of Grassmann manifolds Rewrite the objective function to matrix form { } f(x, Y, Z) = A (X, Y, Z) U, V, W orthogonal = X T A (1) (Y Z) = U T X T A (1) (Y Z) = Y T A (2) (Z X) = V T Y T A (2) (Z X) = Z T A (3) (X Y) = W T Z T A (3) (X Y) = A (XU, Y V, Z W) X XU, Y Y V, Z Z W Only the subspace spanned by X, Y, Z matters!

16 Newton-type optimization on manifold Denote the Grassmann manifold G(n, r) and the tangent space T X at X G(n, r) R n r T X T X = 0 Iterate until convergence 1 compute f 2 compute Hessian or update Hessian 3 X G(n, r) solve H = f 4 Update iterate X: move along geodesic in direction given by X(t) = XV cos(σt)v T + U sin(σt)v T = UΣV T

17 The special case of G(3, 1), i.e. X R 3 and X T X = 1. geodesic curve vector in tangent space of X point X on the manifold

18 in Euclidean space The where B k+1 = B k B ks k s T k B k s T k B ks k s k = x k+1 x k = t k p k y k = f k+1 f k + y k y T k y T k y k On Grassmann manifold the vectors are defined on different points belonging to different tangent spaces.

19 Different ways of parallel transporting vectors X G(n, r), 1, 2 T X and X(t) geodesic path along 1 Parallel transport using global coordinates 2 (t) = T 1 (t) 2

20 Different ways of parallel transporting vectors X G(n, r), 1, 2 T X and X(t) geodesic path along 1 Parallel transport using global coordinates 2 (t) = T 1 (t) 2 we have also 1 = X D 1 and 2 = X D 2 where X basis for T X. Let X(t) be basis for T X(t) Parallel transport using local coordinates 2 (t) = X(t) D 2

21 Parallel transport in local coordinates is in general more efficient All transported have the same coordinate representation in the basis X(t) at all points on the path X(t). + no need to transport the gradient or the Hessian - we need to compute X(t). In global coordinate we compute T k+1 s k = t k T pk (t k )p k T k+1 y k = f k+1 T pk (t k ) f k T pk (t k )H k Tp 1 k (t k ) : T k+1 T k+1 B k+1 = B k B ks k s T k B k s T k B ks k + y k y T k y T k y k

22 The objective function and directional derivative f(x, Y, Z) = 1 2 A (X, Y, Z) 2 F = 1 A (X, Y, Z), A (X, Y, Z) 2 Differentiate along three tangent directions x, y, z df dt = 1 d A (X, Y, Z), A (X, Y, Z) 2 dt = A ( x, Y, Z), A (X, Y, Z) + A (X, y, Z), A (X, Y, Z) + A (X, Y, z ), A (X, Y, Z) = x, A (I, Y, Z), A (X, Y, Z) 1 + y, A (X, I, Z), A (X, Y, Z) 2 + z, A (X, Y, I), A (X, Y, Z) 3

23 The Grassmann Gradient In global coordinates A (Π x, Y, Z), A (X, Y, Z) 1 f = A (X,Π y, Z), A (X, Y, Z) 2 A (X, Y,Π z ), A (X, Y, Z) 3 where Π x = I XX T, Π y = I YY T, Π z = I ZZ T

24 The Grassmann Gradient In global coordinates A (Π x, Y, Z), A (X, Y, Z) 1 f = A (X,Π y, Z), A (X, Y, Z) 2 A (X, Y,Π z ), A (X, Y, Z) 3 where Π x = I XX T, Π y = I YY T, Π z = I ZZ T In local coordinates, X, Y, Z basis for the tangent spaces. A (X, Y, Z), A (X, Y, Z) 1 f = A (X, Y, Z), A (X, Y, Z) 2 A (X, Y, Z ), A (X, Y, Z) 3

25 A (X, Y, Z) Interpretations of the gradient The entries of the gradient are scalars products between slices from the illustrated tensor blocks. Consider A (X, Y, Z), A (X, Y, Z) 1 A (X, Y, Z ) A (X, Y, Z) A (X, Y, Z)

26 f(x) = 1 2 A (X, X, X) 2 F = 1 A (X, X, X), A (X, X, X) 2 The becomes with Π = I XX T If A is symmetric then f = A (Π, X, X), A (X, X, X) 1 + A (X,Π, X), A (X, X, X) 2 + A (X, X,Π), A (X, X, X) 3 f = 3 A (Π, X, X), F 1

27 convergence plots and computation times random tensor approximated with a rank-(5,5,5) tensor RELATIVE NORM OF THE GRADIENT QNG NG CGG HOOI ITERATION # QNG NG CGG HOOI (250 it, rel. norm 5.6e-8) 14 (250 it, rel. norm 2e-10)

28 convergence plots and computation times random tensor approximated with a rank-(10,10,10) tensor RELATIVE NORM OF THE GRADIENT QNG NG CGG HOOI ITERATION # QNG NG CGG HOOI (500 it, rel. norm 1.5e-7) 51 ( 500 it, rel. norm 2.7e-9)

29 convergence plots and computation times symmetric random tensor approximated with a rank-(5,5,5) symmetric tensor RELATIVE NORM OF THE GRADIENT QNG NG HOOI ITERATION # QNG NG HOOI (500 it, rel. norm 1e-6)

30 convergence plots and computation times symmetric random tensor approximated with a rank-(10,10,10) symmetric tensor RELATIVE NORM OF THE GRADIENT QNG NG HOOI ITERATION # QNG NG HOOI (500 it, rel. norm 1e-7)

31 A few remarks and questions Need good starting points, truncated not always good, not always available Need to make efficient implementation of the algorithm The better initial Hessian the better the convergence Good Hessian or rather inverse Hessian is needed at starting point What properties of a tensor determine convergence rates of different algorithms?

32 Presented a brief introduction to tensors and multilinear algebraic. Presented an algebraic framework for solving the best multilinear rank of a tensor on a product of Grassmann manifolds. Both the general and the symmetric tensor problems were addressed. Compare different algorithms and the fast convergence of the QNG algorithm was verified.

33 For Further Reading Lars Eldén and. A Newton Grassmann method for computing the Best Multi-Linear Rank-(r 1, r 2, r 3 ) Approximation of a. submitted to SIAM Journal on Matrix Analysis and Applications, and Lek-Heng Lim Best multilinear rank with quasi-newton method on Grassmannians. soon available as a techreport, 2007.

34 Thank you for your time Questions??, PhD student Department of Mathematics Linköpings Universitet SE Linköping Web Phone +46 (0) Fax +46 (0)