A Geometric Analysis of Subspace Clustering with Outliers

Size: px

Start display at page:

Download "A Geometric Analysis of Subspace Clustering with Outliers"

Ruby Scott
5 years ago
Views:

1 A Geometric Analysis of Subspace Clustering with Outliers Mahdi Soltanolkotabi and Emmanuel Candés Stanford University

2 Fundamental Tool in Data Mining : PCA

3 Fundamental Tool in Data Mining : PCA

4 Subspace Clustering : multi-subspace PCA

5 Subspace Clustering : multi-subspace PCA

6 Subspace Clustering : multi-subspace PCA

7 Subspace Clustering : multi-subspace PCA

8 Subspace Clustering : multi-subspace PCA

9 Index 1 Motivation 2 Problem Formulation 3 Methods 4 Deterministic Model 5 Semi-random Model 6 Fully-random Model 7 With outliers

10 Motivation 1 Motivation 2 Problem Formulation 3 Methods 4 Deterministic Model 5 Semi-random Model 6 Fully-random Model 7 With outliers

11 Motivation Computer Vision Many applications. Figures from various papers/talks. I refer to the interesting lecture by Prof. Ma tomorrow which highlights the relevance of low dimensional structures in computer vision. Just add multiple Tuesday, January 10, 2012 categories :)

Usually, each metabolic disease causes a

12 Motivation Metabolic Screening Goal Detect metabolic disease as early as possible in new borns. concentration of metabolites are tested. Usually, each metabolic disease causes a correlation between the concentration of a specific set of metabolites.

13 Problem Formulation 1 Motivation 2 Problem Formulation 3 Methods 4 Deterministic Model 5 Semi-random Model 6 Fully-random Model 7 With outliers

14 Problem Formulation Problem Formulation N unit norm points in R n on a union of unknown linear subspaces S 1 S 2... S L of unknown dimensions d 1, d 2,..., d L. Also, N 0 uniform outliers. outliers inliers Goal : Without any prior knowledge about the number of subspaces, their orientation or their dimension, 1) identify all the outliers, and 2) segment or assign each data point to a cluster as to recover all the hidden subspaces.

15 Methods 1 Motivation 2 Problem Formulation 3 Methods 4 Deterministic Model 5 Semi-random Model 6 Fully-random Model 7 With outliers

16 Methods Sparse Subspace Clustering Regress one column against other columns. for i = 1,..., N solve min z i l1 subject to x i = Xz i, z ii = 0. =

17 Methods l 1 subspace detection property Definition (l 1 Subspace Detection Property) The subspaces {S l } L l=1 and points X obey the l 1 subspace detection property if and only if it holds that for all i, the optimal solution has nonzero entries only when the corresponding columns of X are in the same subspace as x i.

18 Methods SSC Algorithm Algorithm 1 Sparse Subspace Clustering (SSC) Input: A data set X arranged as columns of X R n N. 1. Solve (the optimization variable is the N N matrix Z) minimize subject to Z 1 XZ = X diag(z) = Form the affinity graph G with nodes representing the N data points and edge weights given by W = Z + Z T. 3. Sort the eigenvalues σ 1 σ 2... σ N of the normalized Laplacian of G in descending order, and set ˆL = N arg max σ i σ i+1. i=1,...,n 1 4. Apply a spectral clustering technique to the affinity graph using ˆL. Output: Partition X 1,...,X ˆL. Subspaces of nearly linear dimension. We prove that in generic settings, SSC can effectively cluster the data even when the dimensions of the subspaces grow almost linearly with the ambient dimension. We are not aware of other literature explaining why this should

19 Methods History of subspace problems and use of l 1 regression Various subspace clustering algorithms proposed by Profs. Ma, Sastry, Vidal, Wright and collaborators.

20 Methods History of subspace problems and use of l 1 regression Various subspace clustering algorithms proposed by Profs. Ma, Sastry, Vidal, Wright and collaborators. Sparse representation in the multiple subspace framework initiated by Prof. Yi Ma and collaborators in the context of face recognition.

21 Methods History of subspace problems and use of l 1 regression Various subspace clustering algorithms proposed by Profs. Ma, Sastry, Vidal, Wright and collaborators. Sparse representation in the multiple subspace framework initiated by Prof. Yi Ma and collaborators in the context of face recognition. Different from subspace clustering because you know the subspaces in advance.

22 Methods History of subspace problems and use of l 1 regression Various subspace clustering algorithms proposed by Profs. Ma, Sastry, Vidal, Wright and collaborators. Sparse representation in the multiple subspace framework initiated by Prof. Yi Ma and collaborators in the context of face recognition. Different from subspace clustering because you know the subspaces in advance. For subspace clustering by Elhamifar and Vidal. Coined the name Sparse Subspace Clustering (SSC).

23 Methods Our goal Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. Limited theoretical guarantees, Huge gap between theory and practice.

24 Methods Our goal Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. Limited theoretical guarantees, Huge gap between theory and practice. In direct analogy with l 1 regression results before compressed sensing results in 2004.

25 Methods Our goal Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. Limited theoretical guarantees, Huge gap between theory and practice. In direct analogy with l 1 regression results before compressed sensing results in Our contribution is to bridge the gap between theory and practice, specifically :

26 Methods Our goal Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. Limited theoretical guarantees, Huge gap between theory and practice. In direct analogy with l 1 regression results before compressed sensing results in Our contribution is to bridge the gap between theory and practice, specifically : Intersecting Subspaces.

27 Methods Our goal Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. Limited theoretical guarantees, Huge gap between theory and practice. In direct analogy with l 1 regression results before compressed sensing results in Our contribution is to bridge the gap between theory and practice, specifically : Intersecting Subspaces. Subspaces of nearly linear dimension.

28 Methods Our goal Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. Limited theoretical guarantees, Huge gap between theory and practice. In direct analogy with l 1 regression results before compressed sensing results in Our contribution is to bridge the gap between theory and practice, specifically : Intersecting Subspaces. Subspaces of nearly linear dimension. Outliers.

29 Methods Our goal Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. Limited theoretical guarantees, Huge gap between theory and practice. In direct analogy with l 1 regression results before compressed sensing results in Our contribution is to bridge the gap between theory and practice, specifically : Intersecting Subspaces. Subspaces of nearly linear dimension. Outliers.

30 Methods Models We will consider three different models regarding the orientation of subspaces and distribution of points on each subspace. Orientation distribution Deterministic Semi-random Fully-random : Random : Deterministic

31 Deterministic Model 1 Motivation 2 Problem Formulation 3 Methods 4 Deterministic Model 5 Semi-random Model 6 Fully-random Model 7 With outliers

32 Deterministic Model Deterministic Model Theorem (M. Soltanolkotabi, E. J. Candes) If µ(x l ) < min i: x i X l r(p l i) (1) for each l = 1,..., L, then the subspace detection property holds. If (1) holds for a given l, then a local subspace detection property holds in the sense that for all x i X l, the solution to the optimization problem has nonzero entries only when the corresponding columns of X are in the same subspace as x i. As long as the point sets are not very similar and the points on each subspace are well distributed SSC works.

33 Deterministic Model Geometric view of l 1 subspace detection

34 Deterministic Model Geometric view on the condition of the theorem

35 Semi-random Model 1 Motivation 2 Problem Formulation 3 Methods 4 Deterministic Model 5 Semi-random Model 6 Fully-random Model 7 With outliers

36 Semi-random Model principal angle and affinity between subspaces Definition The principal angles θ (1) k,l,..., θ(d k dl } k,l between two subspaces S k and S l of dimensions d k and d l, are recursively defined by cos(θ (i) kl ) = max max y S k z S l y T z y l2 z l2 = y T i z i y i l2 z i l2. with the orthogonality constraints y T y j = 0, z T z j = 0, j = 1,..., i 1. Definition The affinity between two subspaces is defined by aff(s k, S l ) = cos 2 θ (1) kl cos2 θ (d k dl ) kl. For random subspaces square of affinity is the Pillai-Bartlett test Statistics.

37 Semi-random Model Semi-random model Assume we have N l = ρ l d l + 1 random points on subspace S l. Theorem (M. Soltanolkotabi, E. J. Candes) Ignoring some log factors, as long as max k : k l aff(s k, S l ) dk < c log ρ l, for each l, (2) the subspace detection property holds with high probability. The affinity can at most be d k and, therefore, our result essentially states that if the affinity is less than c d k, then SSC works. allows for intersection of subspaces.

(z i ) in the form we z i1 z i = Γ z i2.

38 Semi-random Model Four error criterion feature detection error : For each point x i in subspace S li partition the optimal solution of SSC (z i ) in the form we z i1 z i = Γ z i2. z i3 1 N N i=1 (1 z ik i l1 z i l1 ). clustering error : # of misclassified points. total # of points

39 Semi-random Model Four error criterion error in estimating the number of subspaces : This error quantity indicates whether our version of the SSC algorithm can correctly identify the correct number of subspaces ; 0 when it succeeds, 1 when it fails. Singular values of the normalized Laplacian : σ N L < ɛ.

40 Semi-random Model affinity example Choose difficult case of n = 2d, the three basis we choose for each subspace are [ ] [ ] U (1) Id =, U (2) 0d d =, U (3) = 0 d d I d 1 cos(θ 1 ) cos(θ 2 ) cos(θ 3 ) cos(θ d ) sin(θ 1 ) sin(θ 2 ) sin(θ 3 ) sin(θ d ) cosθi α 0 1 i d

41 Semi-random Model feature detection error aff/ d ρ 0

42 Semi-random Model clustering error aff/ d ρ 0

43 Semi-random Model error in estimating the number of subspaces aff/ d ρ 0

44 Fully-random Model 1 Motivation 2 Problem Formulation 3 Methods 4 Deterministic Model 5 Semi-random Model 6 Fully-random Model 7 With outliers

45 Fully-random Model Fully-random model L subspaces, each of dimension d, ρd + 1 points on each subspace. Theorem (M. Soltanolkotabi, E. J. Candés) The subspace detection property holds with high probability as long as where c > c2 (ρ) log ρ 12 log N. d < c n,

46 Fully-random Model Fully-random model L subspaces, each of dimension d, ρd + 1 points on each subspace. Theorem (M. Soltanolkotabi, E. J. Candés) The subspace detection property holds with high probability as long as where c > c2 (ρ) log ρ 12 log N. d < c n, allows for the dimension of the subspaces d to grow almost linearly with the ambient dimension n.

47 Fully-random Model Fully-random model L subspaces, each of dimension d, ρd + 1 points on each subspace. Theorem (M. Soltanolkotabi, E. J. Candés) The subspace detection property holds with high probability as long as where c > c2 (ρ) log ρ 12 log N. d < c n, allows for the dimension of the subspaces d to grow almost linearly with the ambient dimension n. tend n, with d n a constant, ρ = dη with η > 0 the condition reduces to 1 d < 48(η + 1) n

48 Fully-random Model Comparison with previous analysis Condition of Elhamifar and Vidal max cos(θ kl) < 1 k: k l dl max σ min (Y ) for all l = 1,..., L, Y W dl (X (l) ) semi-random model : -Elhamifar and Vidal s condition reduces to cos θ kl < 1 d. -We require aff(s k, S l ) = cos 2 (θ (1) kl ) + cos2 (θ (2) kl ) cos2 (θ (d) kl ) < c d. The left-hand side can be much smaller than d cos θ kl and is, therefore, less restrictive, e.g. cos θ kl < c is sufficient.

49 Fully-random Model Comparison with previous analysis contd. Intersecting subspaces : Two subspaces with an intersection of dimension s. -Elhamifar and Vidal becomes 1 < 1 d, which cannot hold. -Our condition simplifies to cos 2 (θ (s+1) kl fully random model : -Elhamifar and Vidal : d < c 1 n. - We essentially require : d < cn. ) cos 2 (θ (d) kl ) < cd s,

50 Fully-random Model Subspaces Intersect s =1 s =2 s =3 s =4

51 Fully-random Model feature detection error s

52 Fully-random Model clustering error s

53 Fully-random Model error in estimating the number of subspaces s

54 With outliers 1 Motivation 2 Problem Formulation 3 Methods 4 Deterministic Model 5 Semi-random Model 6 Fully-random Model 7 With outliers

55 With outliers subspace clustering with outliers Assume the outliers are uniform at random on the unit sphere. Again, regress one column against other columns. for i = 1,..., N solve min z i l1 subject to x i = Xz i, z ii = 0. consider optimal values z i l 1. Inliers : Intuitively, size of support d, z i l 1 d.

56 With outliers subspace clustering with outliers Assume the outliers are uniform at random on the unit sphere. Again, regress one column against other columns. for i = 1,..., N solve min z i l1 subject to x i = Xz i, z ii = 0. consider optimal values z i l 1. Inliers : Intuitively, size of support d, z i l 1 d. Outliers : Intuitively, size of support n, z i l 1 n.

57 With outliers subspace clustering with outliers Assume the outliers are uniform at random on the unit sphere. Again, regress one column against other columns. for i = 1,..., N solve min z i l1 subject to x i = Xz i, z ii = 0. consider optimal values z i l 1. Inliers : Intuitively, size of support d, z i l 1 d. Outliers : Intuitively, size of support n, z i l 1 n. Therefore, we should expect a gap.

58 With outliers Inliers : Intuitively, size of support d, z i l 1 d. Here, z i l 1 = 1.05.

59 With outliers Inliers : Intuitively, size of support d, z i l 1 d. Here, z i l 1 = Outliers : Intuitively, size of support n, z i l 1 n. Here, z i l 1 = 1.72.

60 With outliers Algorithm for subspace clustering with outliers [M. Soltanolktoabi, E. J. Candes] Algorithm 2 Subspace clustering in the presence of outliers Input: A data set X arranged as columns of X Rn N. 1. Solve minimize Z 1 subject to XZ = X diag(z) = For each i {1,..., N }, declare i to be an outlier iﬀ zi 1 > λ(γ) n.3 3. Apply a subspace clustering to the remaining points. Output: Partition X0, X1,..., XL. Theorem 1.3 Assume there are Nd points to be clustered together with N0 outliers sampled unin 1 on the n 1-dimensional unit sphere (N = N0 + N ). Algorithm 2 detects all of formly d 2 γ at =random n and λ is a threshold ratio function. the outliers with high probability4 as long as N0 < 1 c n e Nd, n where c is a numerical constant. Furthermore, suppose the subspaces are d-dimensional and of arbitrary orientation, and that each contains ρd + 1 points sampled independently and uniformly

61 With outliers Threshold function λ(γ) = 2 π 2 πe 1 γ, if 1 γ e, 1 log γ, if γ e, λ(γ) γ

62 With outliers with outliers Theorem (M. Soltanolkotabi, E. J. Candés) With outlier points uniformly random using the threshold value (1 t) λ(γ) e n, all outliers are identified correctly with high probability. Furthermore, we have the following guarantees in the deterministic and semi-random models. (a) If in the deterministic model, max l,i 1 < (1 t)λ(γ) r(p(x (l) i )) then no real data point is wrongfully detected as an outlier. e n, (3) (b) If in the semi-random model, 2dl max l c(ρ l ) < (1 t) λ(γ) n, (4) log ρ l e then w.h.p. no real data point is wrongfully detected as an outlier.

63 With outliers Comparison with previous analysis our results restricts the number of outliers to : Lerman and Zhang use N 0 < min{ρ c 2n/d, 1 n ec n } N d e lp (X, S 1,..., S L ) = x X ( min dist(x, Sl ) ) p. 1 l L with 0 p 1. They bound on the number of outliers in the semi-random model : N 0 < τ 0 ρd min ( 1, min k l dist(s k, S l ) p /2 p). which is upper bounded by ρd, the typical number of points per subspace. non-convex, no practical method known for solving it.

64 With outliers Segmentation with outliers 8 7 l 1 norm values Proven Threshold Conjuctured Threshold Figure: d = 5, n = 200, ρ = 5, L = 80

65 With outliers References List is not comprehensive! A geometric analysis of subspace clustering. M. Soltanolkotabi and E. J. Candes to appear in Annals of Statistics. Noisy case, in preparation. Joint with E. J. Candes, E. Elhamifar, and R. Vidal. Sparse Subspace Clustering. E. Elhamifar and R. Vidal. Subspace Clustering. Turorial by R. Vidal. Prof. Yi Ma s website for papers on related topics :

66 With outliers Conclusion Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice.

67 With outliers Conclusion Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. In direct analogy with l 1 regression results before compressed sensing results in 2004.

68 With outliers Conclusion Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. In direct analogy with l 1 regression results before compressed sensing results in Our contributions is to bridge the gap between theory and practice, specifically :

69 With outliers Conclusion Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. In direct analogy with l 1 regression results before compressed sensing results in Our contributions is to bridge the gap between theory and practice, specifically : Intersecting Subspaces.

70 With outliers Conclusion Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. In direct analogy with l 1 regression results before compressed sensing results in Our contributions is to bridge the gap between theory and practice, specifically : Intersecting Subspaces. Subspaces of nearly linear dimension.

71 With outliers Conclusion Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. In direct analogy with l 1 regression results before compressed sensing results in Our contributions is to bridge the gap between theory and practice, specifically : Intersecting Subspaces. Subspaces of nearly linear dimension. Outliers.

72 With outliers Conclusion Prior work restrictive, essentially say that the subspaces must be almost independent. Can not fully explain why SSC works so well in practice. In direct analogy with l 1 regression results before compressed sensing results in Our contributions is to bridge the gap between theory and practice, specifically : Intersecting Subspaces. Subspaces of nearly linear dimension. Outliers. Geometric Insights.

73 Thank you With outliers

DD2429 Computational Photography :00-19:00

. Examination: DD2429 Computational Photography 202-0-8 4:00-9:00 Each problem gives max 5 points. In order to pass you need about 0-5 points. You are allowed to use the lecture notes and standard list