Low Rank Representation Theories, Algorithms, Applications. 林宙辰 北京大学 May 3, 2012

Transcription

1 Low Rank Representation Theories, Algorithms, Applications 林宙辰 北京大学 May 3, 2012

2 Outline Low Rank Representation Some Theoretical Analysis Solution by LADM Applications Generalizations Conclusions

3 Sparse Representation Sparse Representation m in jjxjj 0 ; s:t: y = A x: (1) Sparse Subspace Clustering m in jjz i jj 0 ; s:t: x i = X^iz i ; 8i: (2) where X^i = [x 1 ; ; x i 1 ; x i+1 ; ; x n ]. m in jjz jj 0 ; s:t: X = X Z ; diag(z ) = 0: (3) m in jjz jj 1 ; s:t: X = X Z ; diag(z ) = 0: E. Elhamifar and R. Vidal. Sparse Subspace Clustering. CVPR2009. (4)

4 Sparse Representation Construct a graph W = (jz j + j(z ) T j)=2 Normalized cut on the graph E. Elhamifar and R. Vidal. Sparse Subspace Clustering. CVPR2009.

5 Sparse Representation Theorem. Assum e the data is clean and is drawn from independent subspaces, then Z is block diagonal. dim ( P i S i ) = P i dim (S i ): E. Elhamifar and R. Vidal. Sparse Subspace Clustering. CVPR2009.

6 Drawback of SSC Sensitive to noise: no cross validation among coefficients m in jjz jj 1 ; s:t: X = X Z ; diag(z ) = 0: (4) m in jjz i jj 1 ; s:t: x i = X z i ; (z i ) i = 0: (5) E. Elhamifar and R. Vidal. Sparse Subspace Clustering. CVPR2009.

7 Hints from 2D Sparsity Rank is a good measure of 2D sparsity Real data usually lie on low-dim manifolds > 1B dim low-dim subspaces low rank data matrices Low rank high correlation among rows/columns

8 Low Rank Representation m in jjz jj 1 ; s:t: X = X Z ; diag(z ) = 0: (4) m in jjz jj ; s:t: X = X Z : no additional constraint! (6) jjz jj = P j ¾ j (Z ), nuclear norm, a convex surrogate of rank. Liu et al. Robust Subspace Segmentation by Low-Rank Representation, ICML 2010.

9 Low Rank Representation T heorem. A ssum e the data is clean and is draw n from indep endent subspaces, then there exists Z w hich is blo ck diagonal, and the rank of each blo ck equals the dim ension of the corresp onding subspace. Liu et al. Robust Subspace Segmentation by Low-Rank Representation, ICML 2010.

10 Low Rank Representation When there is noise and outliers m in jjz jj + jje jj 2;1 ; s:t: X = X Z + E : (7) where jje jj 2;1 = P i jje :;i jj 2. Liu et al. Robust Subspace Segmentation by Low-Rank Representation, ICML 2010.

11 Low Rank Representation Connection to Robust PCA m in ka k + jje jj 1 ; A ;E s:t: X = A + E ; (8) The clean data is low rank w.r.t. the dictionary I. m in jjz jj + jje jj 2;1 ; s:t: X = X Z + E : (7) The clean data is low rank w.r.t. the dictionary X. Candes, Li, Ma, and Wright, Robust Principal Component Analysis? Journal of the ACM, 2010.

12 Low Rank Representation Generalization m in jjz jj + jje jj 2;1 ; s:t: X = A Z + E : (9) The clean data is low rank w.r.t. the dictionary A. Liu et al. Robust Subspace Segmentation by Low-Rank Representation, ICML 2010.

13 More Theoretical Analysis Closed form solution at noiseless case m in kz k ; Z s:t: X = X Z ; has a unique closed-form optim al solution: Z = V r V T r skinny SV D of X. Shape Interaction Matrix, w here U r r V T r when X is sampled from independent subspaces, Z* is block diagonal, each block corresponding to a subspace is th e min kzk = rank(x): X=XZ S. Wei and Z. Lin. Analysis and Improvement of Low Rank Representation for Subspace segmentation, arxiv: , Liu et al. Robust Recovery of Subspace Structures by Low-Rank Representation, TPAMI 2012.

14 More Theoretical Analysis Closed form solution at general case m in Z kz k ; s:t: X = A Z ; has a unique closed-form optim al solution: Z = A y X. Liu et al., Robust Recovery of Subspace Structures by Low-Rank Representation, TPAMI 2012.

15 Follow-up Work More problems with closed form solutions min Z Speeding up optimization "kzk kxz Xk2 F min A;Z "kzk + 2 kaz Ak2 F kd Ak2 F min "kzk + 1 A;Z 2 ka Xk2 F ; s:t: A = AZ: P. Favaro, R. Vidal and A. Ravichandran, A Closed Form Solution to Robust Subspace Estimation and Clustering, CVPR 2011.

16 Exact Recoverability of LRR T heorem : L et = 3=(7kX k p n). T hen there exists > 0 such that w hen < =, LR R can exactly recover the row space and the colum n supp ort of (Z 0 ; E 0 ): U (U ) T = V 0 V T 0 ; I = I 0 ; w here = ji 0 j=n is the fraction of outliers, U is the colum n space of Z, V T 0 is the row space of Z 0, and I and I 0 is the colum n supp orts of E and E 0, resp ectively. Liu et al., Exact Subspace Segmentation and Outlier Detection by Low-Rank Representation, AISTATS Liu et al., Robust Recovery of Subspace Structures by Low-Rank Representation, TPAMI 2012.

17 Linearized Alternating Direction Model Problem ADM Difficulties min x;y Method (LADM) f(x) + g(y); s:t: A(x) + B(y) = c; where x, y and c could be either vectors or matrices, f and g are convex functions, and A and B are linear mappings. L A (x; y; ) = f(x) + g(y) + h ; A(x) + B(y) yi + 2 ka(x) + B(y) ck2 ; x k+1 = arg min x y k+1 = arg min y L A (x; y k ; k); L A (x k+1 ; y; k); k+1 = k + [A(x k+1 ) + B(y k+1 ) c]: Lin et al., Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation, NIPS 2011.

18 Linearized Alternating Direction Method (LADM) x k+1 = arg min f(x) + x 2 ka(x) + B(y k) c + k= k 2 ; y k+1 = arg min g(y) + y 2 ka(x k+1) + B(y) c + k= k 2 Linearize the quadratic term x k+1 = arg min x = arg min x y k+1 = arg min y Adaptive Penalty f(x) + ha ( k) + A (A(x k ) + B(y k ) c); x x k i + A 2 kx x k k 2 f(x) + A 2 kx x k + A ( k + (A(x k ) + B(y k ) c))=( A)k 2 ; g(y) + B 2 ky y k + B ( k + (A(x k+1 ) + B(y k ) c))=( B)k 2 :

19 Linearized Alternating Direction Method (LADM) Theorem: If f kg is non-decreasing and upper bounded, A > kak 2, and B > kbk 2, then the sequence f(x k ; y k ; k)g generated by LADMAP converges to a KKT point of the model problem. Lin et al., Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation, NIPS 2011.

20 Applying LADM to LRR LRR Further Acceleration Technique O(n 3 ) O(rn 2 ) min Z;E kzk + kek 2;1 ; s:t: X = XZ + E: Lin et al., Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation, NIPS 2011.

21 Experiments Table 1: Comparison among APG, ADM, LADM, standard LADMAP and accelerated LADMAP (denoted as LADMAP(A)) on the synthetic data. For each quadruple (s, p, d, ~r), the LRR problem, with ¹ = 0:1, was solved for the same data using di erent algorithms. We present typical running time (in 10 3 seconds), iteration number, relative error (%) of output solution ( ^E; ^Z) and the clustering accuracy (%) of tested algorithms, respectively. Size (s, p, d, ~r) Method Time Iter. (10, 20,200, 5) (15, 20,300, 5) (20, 25, 500, 5) (30, 30, 900, 5) k^z Z 0 k kz 0 k k ^E E 0 k ke 0 k Acc. APG ADM LADM LADMAP LADMAP(A) APG ADM LADM LADMAP LADMAP(A) APG ADM LADM LADMAP LADMAP(A) APG ADM LADM N.A. N.A. N.A. N.A. N.A. LADMAP LADMAP(A)

22 Applications of LRR Image segmentation min Z;E kzk + kek 2;1 ; s:t: X = XZ + E: B. Cheng et al. Multi-task Low-rank Affinity Pursuit for Image Segmentation, ICCV 2011.

23 Saliency detection Applications of LRR m in kz k + ke k 2;1 ; Z ;E s:t: X = X Z + E : Lang et al. Saliency Detection by Multitask Sparsity Pursuit. IEEE TIP 2012.

24 Generalizations of LRR LRR with clean data min D ;Z;E kz k + ke k 2;1 ; s:t: D = D Z; X = D + E : m in D ;Z ;E kz k + ke k 2 F ; s:t: D = D Z ; X = D + E : S. Wei and Z. Lin. Analysis and Improvement of Low Rank Representation for Subspace segmentation, arxiv: , P. Favaro, R. Vidal and A. Ravichandran, A Closed Form Solution to Robust Subspace Estimation and Clustering, CVPR 2011.

25 Latent LRR Generalizations of LRR To address insufficient sample problem (X = [X ; X H ]Z O ;H =) X = X Z + LX ) m in Z ;L ;E kz k + klk + jje jj 2;1 ; s:t: X = X Z + LX + E : Liu and Yan. Latent Low-Rank Representation for Subspace Segmentation and Feature Extraction, ICCV 2011.

26 Generalizations of LRR Fixed Rank Representation (FRR) To address insufficient sample problem Subspace clustering: using m in Z ; ~ Z ;E Feature extraction: by m in Z ; ~ Z ;E ~Z kz ~ Z k 2 F + jje jj 2;1 ; s:t: X = X Z + E ; rank( ~ Z ) <= r: y = ~ Z x kz ~ Z k 2 F + jje jj 2;1 ; s:t: X = Z X + E ; rank( ~ Z ) <= r: Liu et al., Fixed-Rank Representation for Unsupervised Visual Learning, CVPR 2012.

27 Generalizations of LRR Semi-supervised learning min Z;E kz k + kz k 1 + ke k 2;1 ; s:t: X = AZ + E ; Z 0: propogate labels on the graph with weights (Z + (Z ) T )=2. L. Zhuang et al. Non-Negative Low Rank and Sparse Graph for Semi-Supervised Learning, CVPR 2012.

28 Kernel LRR Generalizations of LRR To address nonlinear multi-manifold segmentation m in Z ká(x ) Á(X )Z k 2 F + jjz jj : min Z X (z T i K (X ; X )z i 2K (x i ; X )z i + K (x i ; x i )) + jjz jj : i Kernel trick: há(x); Á(y)i = K (x; y) Wang et al., Structural Similarity and Distance in Learning, Annual Allerton Conf. Communication, Control and Computing 2011.

29 Thanks! m in kz k + ke k 2;1 ; Z ;E s:t: X = X Z + E : LRR = SSC + RPCA