Winter Conference in Statistics Compressed Sensing. LECTURE #1-2 Motivation & geometric insights

Transcription

1 Winter Conference in Statistics 2013 Compressed Sensing LECTURE #1-2 Motivation & geometric insights Prof. Dr. Volkan Cevher LIONS/Laboratory for Information and Inference Systems

2 Major trends 160MP higher resolution / denser sampling x large numbers of sensors x increasing # of modalities / mobility Motivation: solve bigger / more important problems decrease acquisition times / costs entertainment / new consumer products

3 Problems of the current paradigm Sampling at Nyquist rate expensive / difficult Data deluge communications / storage Sample then compress inefficient / impossible / not future proof

4 Recommended for you: A more familiar example Recommender systems observe partial information ratings clicks purchases compatibilities

5 Recommended for you: A more familiar example Recommender systems observe partial information The Netflix problem ratings clicks purchases compatibilities from approx. 100,000,000 ratings predict 3,000,000 ratings movies x users how would you automatically predict?

6 Recommended for you: A more familiar example Recommender systems observe partial information The Netflix problem ratings clicks purchases compatibilities from approx. 100,000,000 ratings predict 3,000,000 ratings movies x users how would you automatically predict? what is it worth?

7 Theoretical set-up Matrix completion for Netflix users movies

8 Theoretical set-up Matrix completion for Netflix users movies Mathematical underpinnings: compressive sensing observations (adversarial) perturbations linear (sampling) operator CS: when we have less samples than the ambient dimension

9 Linear Inverse Problems Myriad applications involve linear dimensionality reduction deconvolution to data mining compression to compressive sensing geophysics to medical imaging [Baraniuk, C, Wakin 2010; Carin et al. 2011]

10 Linear Inverse Problems Challenge:

11 Linear Inverse Problems Deterministic Probabilistic Prior sparsity distribution Metric likelihood/ posterior

12 Deterministic Low-Dimensional Models

13 Sparse representations Sparse signal only K out of N coordinates nonzero support: sorted index

14 Sparse representations Sparse signal only K out of N coordinates nonzero in an appropriate representation Sparse representations sparse transform coefficients Basis representations Wavelets, DCT Frame representations Gabor, curvelets, shearlets Other dictionary representations

15 Sparse representations Sparse signal: only K out of N coordinates nonzero Sparse representations: sparse transform coefficients A fundamental impact:

16 Sparse representations Sparse signal: only K out of N coordinates nonzero Sparse representations: sparse transform coefficients A fundamental impact:

17 Sparse representations Sparse signal: only K out of N coordinates nonzero Sparse representations: sparse transform coefficients A fundamental impact:

18 Sparse representations Sparse signal: only K out of N coordinates nonzero Sparse representations: sparse transform coefficients A fundamental impact: becomes effectively low dimensional* *: If we knew the locations of the coefficients. More on this later.

19 Low-dimensional signal models Information level: pixels large wavelet coefficients (blue = 0) sparse signals low-rank matrices nonlinear models

20 Low-dimensional signal models These lectures Low-dimensional models based on linear representations sparse signals low-rank matrices nonlinear models

21 Linear representation of low-dimensional models A key notion in sparse representation synthesis of the signal using a few vectors A slightly different mathematical formalism for generalization Synthesis model: i.e., linear (positive) combination of elements from an atomic set [Chandrasekaran et al. 2010]

22 Linear representation of low-dimensional models A key notion in sparse representation synthesis of the signal using a few vectors Sparse representations via the atomic formulation Example:

23 Linear representation of low-dimensional models Basic definitions on low-dimensional atomic representations

24 Linear representation of low-dimensional models Basic definitions on low-dimensional atomic representations : convex hull of atoms in A

25 Linear representation of low-dimensional models Basic definitions on low-dimensional atomic representations : convex hull of atoms in A atomic ball

26 Linear representation of low-dimensional models Basic definitions on low-dimensional atomic representations : convex hull of atoms in A : atomic norm* *: requires A to be centrally symmetric

27 Linear representation of low-dimensional models Basic definitions on low-dimensional atomic representations : convex hull of atoms in A : atomic norm* *: requires A to be centrally symmetric

28 Linear representation of low-dimensional models Basic definitions on low-dimensional atomic representations : convex hull of atoms in A : atomic norm* Alternative: *: requires A to be centrally symmetric

29 Linear representation of low-dimensional models Examples with easy forms: sparse vectors low-rank matrices * *symmetric matrices binary vectors

30 Linear representation of low-dimensional models Examples with easy forms: sparse vectors Examples with no-so-easy forms: A : infinite set of unit-norm rank-one tensors low-rank matrices A : finite (but large) set of permutation matrices A : infinite set of orthogonal matrices A : infinite set of matrices constrained by eigenvalues A : infinite set of measures binary vectors A : finite (but large) set of cut matrices [Chandrasekaran et al. 2010]

31 Linear representation of low-dimensional models Pop-quiz: What is?

32 Linear representation of low-dimensional models Pop-quiz: HINT: What is?

33 Linear representation of low-dimensional models Pop-answer: What is?

34 Towards algorithms: a geometric perspective Other key concepts:

35 Towards algorithms: a geometric perspective Other key concepts:

36 Towards algorithms: a geometric perspective Other key concepts: Tangent cone is the set of descent directions where you do not increase the atomic norm.

37 Towards algorithms: a geometric perspective Other key concepts: Tangent cone is the set of descent directions where you do not increase the atomic norm.

38 Towards algorithms: a geometric perspective

39 Towards algorithms: a geometric perspective

40 Towards algorithms: a geometric perspective Consider the criteria:

41 Towards algorithms: a geometric perspective Consider the criteria:

42 Towards algorithms: a geometric perspective Consider the criteria:

43 Towards algorithms: a geometric perspective Consider the criteria:

44 Towards algorithms: a geometric perspective Key observation:

45 Towards algorithms: a geometric perspective How about noise?

46 Towards algorithms: a geometric perspective How about noise? Stability assumption:

47 Towards algorithms: a geometric perspective How about noise? Stability assumption:

48 Towards algorithms: a geometric perspective How about noise? Stability assumption: want epsilon large to minimize overlap between and For this 2D example: Matlab notation

49 Towards algorithms: a geometric perspective How about noise? Stability assumption:

50 Towards algorithms: a geometric perspective Can we guarantee the following?* *without knowing

51 Towards algorithms: a geometric perspective Can we guarantee the following?* YES: with randomized measurements! Gordon s Minimum Restricted Singular Values Theorem has a probabilistic characterization. probabilistic deterministic [Gordon 1988] *without knowing

52 Towards algorithms: a geometric perspective Can we guarantee the following?* Gordon s Minimum Restricted Singular Values Theorem has a probabilistic characterization. Key concept: width of the tangent cone! *without knowing

53 Towards algorithms: a geometric perspective Can we guarantee the following?* Gordon s Minimum Restricted Singular Values Theorem has a probabilistic characterization. *without knowing

54 Towards algorithms: a geometric perspective Can we guarantee the following?* Gordon s Minimum Restricted Singular Values Theorem has a probabilistic characterization. *without knowing

55 Towards algorithms: a geometric perspective Can we guarantee the following?* Gordon s Minimum Restricted Singular Values Theorem has a probabilistic characterization. *without knowing

56 Towards algorithms: a geometric perspective Key observation:

57 Towards algorithms: a geometric perspective How about noise? Stability assumption:

58 Towards algorithms: a geometric perspective Can we guarantee the following?* Gordon s Minimum Restricted Singular Values Theorem has a probabilistic characterization. *without knowing

59 Towards algorithms: a geometric perspective Can we guarantee the following?* *without knowing 1-sparse and 1-random measurement

60 Towards algorithms: a geometric perspective Can we guarantee the following?* *without knowing 1-sparse and 1-random measurement

61 Towards algorithms: a geometric perspective Can we guarantee the following?* *without knowing 1-sparse and 1-random measurement

62 Towards algorithms: a geometric perspective Can we guarantee the following?* A projected 6D hypercube with 64 vertices Blessing-of-dimensionality!

63 Towards algorithms: a geometric perspective Pop-quiz: What is the probability that we can determine a 2-sparse x* with 1-random measurement?

64 Towards algorithms: a geometric perspective Pop-answer: Tangent cone is too wide! Need at least 2 measurements!

65 Take home messages [Chandrasekaran et al. 2010] convex polytope <> atomic norm geometry (and algebra) of representations in high dimensions geometric perspective <> convex criteria convex optimization algorithms in high dimensions tangent cone width <> # of randomized samples probabilistic concentration-of-measures in high dimensions

66 Back to the initial example Matrix completion for Netflix movies x users users movies What is low-rank?

67 Back to the initial example Matrix completion for Netflix movies x users users movies What does the simple low-rank assumption buy? quite a lot of extrapolation power!

68

69 three and do this fast with theoretical guarantees

70 Sampling/sketching design scene single photon detector +Coding theory +Theoretical computer science +Learning theory +Databases random pattern on DMD array DMD DMD image reconstruction or processing Structured random matrices 1-bit CS expanders & extractors

71 Structured recovery Sparsity +Theoretical computer science +Learning theory +Optimization +Databases sorted index Sparse vector only K out of N coordinates nonzero

72 Structured recovery Sparsity +Theoretical computer science +Learning theory +Optimization +Databases sorted index Structured sparse vector only certain K out of N coordinates nonzero

73 Structured recovery Structured sparsity +Theoretical computer science +Learning theory +Optimization +Databases + requires smaller sketches + enhanced recovery + faster recovery support of the solution <> modular approximation problem integer linear program matroid structured sparse models clustered /diversified sparsity models tightly connected with max-cover, binpacking, knapsack problems Recovery with low-dimensional models, including low-rank

74 Quantum tomography Quantum state estimation +Theoretical computer science +Databases +Information theory +Optimization 1 a state of n possibly-entangled qubits takes ~2 n bits to specify, even approximately 0 Recovery with rank and trace constraints with M=O(N) 1. Create Pauli measurements (semi-random) 2. Estimate Tr(Φ i ρ) for each 1 i M 3. Find any hypothesis state σ st Tr(Φ i σ) Tr(Φ i ρ) for all 1 i M Huge dimensional problem! (desperately) need scalable algorithms also need theory for perfect density estimation

75 Learning theory and methods +Learning theory +Optimization A fundamental problem: +Information theory +Theoretical computer science given learn a mapping Our interest <> non-parametric functions graphs (e.g., social networks) dictionary learning Rigorous foundations <> sample complexity approximation guarantees tractability Key tools <> sparsity/low-rankness submodularity smoothness

76 Compressible priors +Learning theory +Statistics +Information theory Goal: seek distributions whose iid realizations can be well-approximated as sparse Definition: relative k-term approximation: sorted index

77 Compressible priors +Learning theory +Information theory Goal: seek distributions whose iid realizations can be well-approximated as sparse Classical: New: sorted index

78 Compressible priors +Learning theory +Statistics +Information theory Goal: seek distributions whose iid realizations can be well-approximated as sparse Motivations: deterministic embedding scaffold for the probabilistic view analytical proxies for sparse signals learning (e.g., dim. reduced data) algorithms (e.g., structured sparse) information theoretic (e.g., coding) lots of applications in vision, image understanding / analysis

79 References M. Afonso, J. Bioucas-Dias, M. Figueiredo, Fast image recovery using variable splitting and constrained optimization, IEEE Transactions on Image Processing, vol. 19, H. Attouch, J. Bolte, P. Redont, A. Soubeyran, Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Lojasiewicz inequality, Math. Oper. Research, 2010 O. Axelsson, Iterative Solution Methods, Cambridge University Press, R. Baraniuk, V. Cevher, M. Duarte, C. Hegde, Model-based compressive sensing, IEEE Transactions on Information Theory, vol. 56, R. Baraniuk, M. Davenport, R. de Vore, M. Wakin, A Simple Proof of the Restricted Isometry Property for Random Matrices, Constructive Approximation, R. Baraniuk, V. Cevher, M. Wakin, Low-dimensional models for dimensionality reduction and signal recovery: A geometric perspective, Proceedings of the IEEE, vol. 98, J. Barzilai and J. Borwein, Two point step size gradient methods, IMA Journal of Numer. Anal., vol. 8, R. Basri, D. Jacobs, Lambertian Reflectance and Linear Subspaces, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, A. Beck, M. Teboulle, A fast iterative shrinkage-thresholding algorithm for linear inverse problems, SIAM Journal on Imaging Science, vol. 2, J. Bioucas-Dias, M. Figueiredo, A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration, IEEE Transactions on Image Processing, vol. 16, P. Boufounos, R. Baraniuk, 1-bit compressed sensing, Proceedings of the Conference on Information Science and Systems, Princeton, C. Boutsidis, M. Mahoney, P. Drineas, An improved approximation algorithm for the column subset selection problem, Proc. 20th Annual ACM/SIAM Symposium on Discrete Algorithms, New York, NY, 2008.

80 References P. Bühlmann, S. van der Geer, Statistics for High-Dimensional Data, Springer, E. Candès, The restricted isometry property and its implications for compressed sensing, Comptes Rendus Mathematique, vol. 346, E. Candès and B. Recht, Exact matrix completion via convex optimization, Foundations of Computational Mathematics, vol. 9, L. Carin, R. Baraniuk, V. Cevher, D. Dunson, M. Jordan, G. Sapiro, M. Wakin, Learning low-dimensional signal models, IEEE Signal Processing Magazine, vol. 28, V. Cevher, An ALPS view of sparse recovery, Proc. ICASSP, V. Chandrasekaran, B. Recht, P. Parrilo, A. Willsky, The convex geometry of linear inverse problems, submitted, R. Chartrand, W. Yin, Iteratively reweighted algorithms for compressive sensing, Proc. ICASSP, 2008 S. Chen, D. Donoho, M. Saunders, Atomic decomposition by Basis Pursuit, SIAM Review, vol. 43, P. Combettes, V. Wajs, Signal recovery by proximal forward-backward splitting, SIAM Journal Multiscale Modeling and Simulation, vol. 4, J. Eckstein, D. Bertsekas, On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators, Mathematical Programming, vol. 5, M. Figueiredo and J. Bioucas-Dias, Restoration of Poissonian images using alternating direction optimization, IEEE Transactions on Image Processing, vol. 19, M. Figueiredo, R. Nowak, S. Wright, Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems, IEEE Journal of Selected Topics in Signal Processing, vol. 1, 2007.

81 References D. Gabay, B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite-element approximations, Computers and Mathematics with Application, vol. 2, R. Glowinski, A. Marroco, Sur l approximation, par elements finis d ordre un, et la resolution, par penalisation-dualité d une classe de problemes de Dirichlet non lineares, Rev. Française d Automatique, Y. Gordon, On Milman's inequality and random subspaces which escape through a mesh in R n, in Geometric Aspects of Functional Analysis, Springer, E. Hale, W. Yin, Y. Zhang, Fixed-point continuation for l1-minimization: Methodology and convergence, SIAM Journal on Optimization, vol. 19, N. Halko, P.-G. Martinsson, J. Tropp, Finding structure with randomness: stochastic algorithms for constructing approximate matrix decompositions, SIAM Review, vol. 53, C. Hegde, M. Duarte, V. Cevher, Compressive sensing recovery of spike trains using a structured sparsity model, Proceedings of SPARS'09, Saint-Malo, France, M. Hestenes, Multiplier and gradient methods, Journal of Optimazion Theory andapplications, vol. 4, A. Kyrillidis, V. Cevher, Recipes for hard thresholding methods, Tech. Rep., EPFL, J. Lee, V. Mirrokni, V. Nagarajan, M. Sviridenko, Non-monotone submodular maximization under matroid and knapsack constraints, Proc. 41st Annual ACM Symposium on Theory of Computing, Bethesda, MD, A. Lewis, J. Malick, Alternating projections on manifolds, Math. of Operations Research, vol. 33, D. Lorenz, Constructing test instances for basis pursuit denoising, submitted, N. Meinshausen, P. Bühlmann, High-dimensional graphs and variable selection with the lasso, The Annals of Statistics, vol. 34, pp , J.-J. Moreau, Proximité et dualité dans un espace hilbertien, Bull. Soc. Mathematiques de France, vol. 93, 1965.

82 References G. Nemhauser, L. Wolsey, Integer and combinatorial optimization, Wiley, S. Osher, M. Burger, D. Goldfarb, J. Xu, W. Yin, An iterative regularization method for total variation-based image restoration, SIAM Journal on Multiscale Modeling and Simulation, vol. 4, Y. Plan, Compressed sensing, sparse approximation, low-rank matrix estimation, PhD Thesis, Caltech, 2011 M. Powell, A method for nonlinear constraints in minimization problems, in Optimization, Academic Press, H. Raguet, J. Fadili, G. Peyré, Generalized Forward-Backward splitting, Tech. report, Hal , S. Setzer, G. Steidl, T. Teuber, Deblurring Poissonian images by split Bregman techniques, Journal of Visual Communication and Image Representation, K.-C. Toh, S. Yun, An accelerated proximal gradient algorithm for nuclear norm regularized least squares Problems, Pacific Journal of Optimization, vol. 6, A. Waters, A. Sankaranarayanan, R. Baraniuk, SpaRCS: recovering low-rank and sparse matrices from compressive measurements, Neural Information Processing Systems, S. Wright, R. Nowak, M. Figueiredo, Sparse reconstruction by separable approximation, IEEE Transactions on Signal Processing, vol. 57, W. Yin, S. Osher, D. Goldfarb, J. Darbon, Bregman iterative algorithms for l1-minimization with applications to compressed sensing, SIAM Journal on Imaging Science, vol. 1, T. Zhou, D. Tao, Godec: randomized low-rank & sparse matrix decomposition in noisy case, Proc. International Conference on Machine Learning, Bellevue, WA, 2011.