An incomplete survey of matrix completion. Benjamin Recht Department of Computer Sciences University of Wisconsin-Madison

Transcription

1 An incomplete survey of matrix completion Benjamin Recht Department of Computer Sciences University of Wisconsin-Madison

2 Abstract Setup: Matrix Completion M = M ij known for black cells M ij unknown for white cells Rows index movies Columns index users How do you fill in the missing data? M = L R * k x n k x r r x n kn entries r(k+n) entries

3 Low-rank Matrix Completion PROBLEM: Find the matrix of lowest rank has the specified entries When is this problem easy? Which algorithms? Which sampling sets? Which low-rank matrices?

4 Which algorithm? PROBLEM: Find the matrix of lowest rank that satisfies/approximates the underdetermined linear system (X) =y : R k n! R m minimize rank(x) subject to (X) =y NP-HARD: Reduce to MAXCUT Hard to approximate Exact algorithms are awful

5 Which Algorithm? Affine Rank Minimization: minimize rank(x) subject to (X) =y Convex Relaxation: minimize kxk = P k i(x) i=1 subject to (X) =y Nuclear Norm Heursistic. Proposed by Fazel (2002). Nuclear norm is the numerical rank in numerical analysis The trace heuristic from controls if X is p.s.d.

6 First theory result (X) =y : R k n! R m If m > c 0 r(k+n-r)log(kn), the heuristic succeeds for most maps Φ. Recht, Fazel, and Parrilo Number of measurements c 0 r(k+n-r) log(kn) constant intrinsic dimension ambient dimension Approach: Show that a random Φ is nearly an isometry on the manifold of low-rank matrices. Stable to noise in measurement vector y and returns as good an answer as a truncated SVD of the true X.

7 If we can choose the samples... Generically, first r rows and r columns are sufficient: M = apple A C [FKV98, DKM03, etc.]: sample proportional to norms of columns. Low-rank matrix approximations. B CA 1 B If we can t choose the samples... Most sets with more than 2rnβ log(n) entries have at least one entry for every row and column, the rowcolumn graph is connected. [AM04]: random sampling sufficient to obtain an additive error approximation to M.

8 Bounds for Matrix Completion Suppose X is k x n (k n) has rank r and has row and column spaces with coherence bounded above by µ. Then the nuclear norm heuristic recovers X from most subsets of entries Ω with cardinality at least Candès and Recht [CT09] stronger assumptions special case extensions: >C 0 nlog(n) [KMO09] rank = o(1), σ1/σr bounded 32µr(n + k) log 2 (2n) [GLFBE09,G09,R09]

9 Incoherence is necessary X = Any subset of entries that misses the (1,1) component tells you nothing! X = Still need to see the entire first row Want each entry to provide nearly the same amount of information

10 RNLA: matrix is an oracle, summarize the matrix, lift the sketch to the full matrix MC: entries are provided in advance, solve a convex program subject to agreeing with the observations Get to choose your samples Assumptions on matrix RNLA YES NO MC NO YES Optimal Rates YES Õ(YES) Robustness A lot Some Fast algorithms YES YES If you can control your sampling set at all do it. When do we ever get uniform samples in reality? Why bother with matrix completion at all? Perhaps there s something something bigger going on.

11 Linear Inverse Problems Find me a solution of y = x m x n, m<n Of the infinite collection of solutions, which one should we pick? Leverage structure: Sparsity Rank Smoothness Symmetry How do we design algorithms to solve underdetermined systems problems with priors?

12 Sparsity 1-sparse vectors of Euclidean norm 1 1 Convex hull is the unit ball of the l 1 norm

13 x 2 Φx=y minimize kxk 1 subject to x = y x 1 Compressed Sensing: Candes, Romberg, Tao, Donoho, Tanner, Etc...

14 Rank 2x2 matrices plotted in 3d rank 1 x 2 + z 2 + 2y 2 = 1 Convex hull: kxk = X i i(x)

15 2x2 matrices plotted in 3d kxk = X i i(x) Nuclear Norm Heuristic Fazel R, Fazel, and Parillo 2007 Rank Minimization/Matrix Completion

16 Integer Programming Integer solutions: all components of x are ±1 (-1,1) (1,1) Convex hull is the unit ball of the l 1 norm (-1,-1) (1,-1)

17 x 2 Φx=y minimize kxk 1 subject to x = y x 1 Donoho and Tanner 2008 Mangasarian and Recht

18 Parsimonious Models rank model weights atoms Search for best linear combination of fewest atoms rank = fewest atoms needed to describe the model

19 Union of Subspaces X has structured sparsity: linear combination of elements from a set of subspaces {Ug}. Atomic set: unit norm vectors living in one of the Ug Permutations and Rankings X a sum of a few permutation matrices Examples: Multiobject Tracking, Ranked elections, BCS Convex hull of permutation matrices: doubly stochastic matrices.

20 Moments: convex hull of of [1,t,t 2,t 3,t 4,...], t T, some basic set. System Identification, Image Processing, Numerical Integration, Statistical Inference Solve with semidefinite programming Cut-matrices: sums of rank-one sign matrices. Collaborative Filtering, Clustering in Genetic Networks, Combinatorial Approximation Algorithms Approximate with semidefinite programming Low-rank Tensors: sums of rank-one tensors Computer Vision, Image Processing, Hyperspectral Imaging, Neuroscience Approximate with alternating leastsquares

21 Atomic Norms Given a basic set of atoms,, define the function A kxk A =inf{t >0 : x 2 tconv(a)} A When is centrosymmetric, we get a norm kxk A =inf{ X a2a c a : x = X a2a c a a} IDEA: minimize subject to kzk A z = y When does this work? How do we solve the optimization problem?

22 Tangent Cones Set of directions that decrease the norm from x form a cone: T A (x) ={d : kx + dk A applekxk A for some > 0} y = z x minimize subject to kzk A z = y T A (x) {z : kzk A applekxk A } x is the unique minimizer if the intersection of this cone with the null space of Φ equals {0}

23 Mean Width Support Function: S C (d) =sup x2c d 0 x d 0 x d 0 x S C (d)+s C ( d) measures width of C when projected onto span of d. mean width: w(c) = Z S n 1 S C (u)du

24 R n When does a random subspace, U in, intersect a convex cone C at the origin? Gordon (1988): with high probability if where codim(u) nw(c \ S n 1 ) 2 w(c \ S n 1 )= Z S n 1 S C (u)du is the mean width. Corollary: For inverse problems, if Φ is a random Gaussian matrix with m rows, need for exact recovery of x. m nw(t A (x) \ S n 1 ) 2

25 Duality w(c \ S n ) = E u apple E u 2 4 max v2c 2 kvk=1 3 hv, ui5 3 4 max hv, ui5 v2c kvkapple1 = E u apple min w2c ku wk C is the polar cone. C = {w : hw, zi apple0 8 z 2 C} T A (x) = N A (x) N A (x) is the normal cone. Equal to the cone induced by the subdifferential of the atomic norm at x. N A (x)

26 Rates Hypercube: m n/2 Sparse Vectors, n vector, sparsity s m 2s log n s + 5s 4 Block sparse, M groups (possibly overlapping), maximum group size B, k active groups m k p 2 log (M k)+ p B 2 + kb Low-rank matrices: n1 x n2, (n1<n2), rank r m 3r(n 1 + n 2 r)

27 General Cones Theorem: Let C be a nonempty cone with polar cone C*. Suppose C* subtends normalized solid angle µ. Then w(c) apple 3 s log 4 µ Corollary: For a vertex-transitive (i.e., symmetric ) polytope with p vertices, O(log p) Gaussian measurements are sufficient to recover a vertex via convex optimization. For n x n permutation matrix: m = O(n log n) For n x n cut matrix: m = O(n)

28 Atomic Norm Minimization IDEA: minimize subject to kzk A z = y Generalizes existing, powerful methods Rigorous formula for developing new analysis algorithms Precise, tight bounds on number of measurements needed for model recovery One algorithm prototype for all data-mining applications

29 Extensions Width Calculations for more general structures (clustering matrices, spectrum estimation, system identification) Recovery bounds for structured measurement matrices (application specific) Understanding of the loss due to convex relaxation and norm approximation Scaling generalized shrinkage algorithms to massive data sets

30 Acknowledgements See: for all references Results developed in collaboration with Emmanuel Candes, Venkat Chandrasekaran, Maryam Fazel, Babak Hassibi, Pablo Parrilo, Weiyu Xu, and Alan Willsky.