Short-course Compressive Sensing of Videos

Transcription

1 Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012 Organizers: Richard G. Baraniuk Mohit Gupta Aswin C. Sankaranarayanan Ashok Veeraraghavan

2 Part 2: Compressive sensing Motivation, theory, recovery Linear inverse problems Sensing visual signals Compressive sensing Theory Hallmark Recovery algorithms Model-based compressive sensing Models specific to visual signals

3 Linear inverse problems

4 Linear inverse problems Many classic problems in computer can be posed as linear inverse problems Notation Signal of interest Observations x 2 R N y 2 R M measurement matrix Measurement model Problem definition: given y = x + e measurement noise y, recover x

5 Linear inverse problems y = x + e x 2 R N y 2 R M Problem definition: given y, recover x Scenario 1 We can invert the system of equations ˆx = M N 1 y Focus more on robustness to noise via signal priors

6 Linear inverse problems y = x + e x 2 R N y 2 R M Problem definition: given y, recover x Scenario 2 Measurement matrix has a (N-M) dimensional null-space Solution is no longer unique Many interesting vision problem fall under this scenario Key quantity of concern: Under-sampling ratio M/N

7 Image super-resolution Low resolution input/observation 128x128 pixels

8 Image super-resolution y = x + e 2x super-resolution y (1,1) =(x 1,1 + x 1,2 + x 2,1 + x 2,2 ) /4

9 Image super-resolution y = x + e 4x super-resolution y (1,1) =(x 1,1 + x 1, x 4,3 + x 4,4 ) /16

10 Image super-resolution y = x + e Super-resolution factor D Under-sampling factor M/N = 1/D 2 General rule: The smaller the under-sampling, the more the unknowns and hence, the harder the superresolution problem

11 Many other vision problems Affine rank minimizaiton, Matrix completion, deconvolution, Robust PCA Image synthesis Infilling, denoising, etc. Light transport Reflectance fields, BRDFs, Direct global separation, Light transport matrices Sensing

12 Sensing visual signals

13 High-dimensional visual signals Reflection Fog Volumetric scattering Human skin Sub-surface scattering Refraction Electron microscopy Tomography

14 Adelson and Bergen (91) The plenoptic function Collection of all variations of light in a scene time (1D) space (3D) spectrum (1D) angle (2D) Different slices reveal different scene properties

15 The plenoptic function time (1D) space (3D) angle (2D) spectrum (1D) High-speed cameras Hyper-spectral imaging Lytro light-field camera

16 Sensing the plenoptic function High-dimensional 1000 samples/dim == 10^21 dimensional signal Greater than all the storage in the world Traditional theories of sensing fail us!

17 Resolution trade-off Key enabling factor: Spatial resolution is cheap! Commercial cameras have 10s of megapixels One idea is the we trade-off spatial resolution for resolution in some other axis

18 Spatio-angular tradeoff [Ng, 2005]

19 Spatio-angular tradeoff [Levoy et al. 2006]

20 Spatio-temporal tradeoff Stagger pixel-shutter within each exposure [Bub et al., 2010]

21 Spatio-temporal tradeoff Rearrange to get high temporal resolution video at lower spatialresolution [Bub et al., 2010]

22 Resolution trade-off Very powerful and simple idea Drawbacks Does not extend to non-visible spectrum 1 Megapixel SWIR camera costs k Linear and global tradeoffs With today s technology, cannot obtain more than 10x for video without sacrificing spatial resolution completely

23 Compressive sensing

24 Sense by Sampling sample

25 Sense by Sampling sample too much data!

26 Sense then Compress sample compress JPEG JPEG2000 decompress

27 Sparsity pixels large wavelet coefficients (blue = 0)

28 Sparsity pixels large wavelet coefficients (blue = 0) wideband signal samples frequency large Gabor (TF) coefficients time

29 Concise Signal Structure Sparse signal: only K out of N coordinates nonzero sorted index

30 Concise Signal Structure Sparse signal: only K out of N coordinates nonzero model: union of K-dimensional subspaces aligned w/ coordinate axes sorted index

31 Concise Signal Structure Sparse signal: only K out of N coordinates nonzero model: union of K-dimensional subspaces Compressible signal: sorted coordinates decay rapidly with power-law power-law decay sorted index

32 Concise Signal Structure Sparse signal: only K out of N coordinates nonzero model: union of K-dimensional subspaces Compressible signal: sorted coordinates decay rapidly with power-law model: ball: power-law decay sorted index

33 What s Wrong with this Picture? Why go to all the work to acquire N samples only to discard all but K pieces of data? sample compress decompress

34 What s Wrong with this Picture? linear processing linear signal model (bandlimited subspace) nonlinear processing nonlinear signal model (union of subspaces) sample compress decompress

35 Compressive Sensing Directly acquire compressed data via dimensionality reduction Replace samples by more general measurements compressive sensing recover

36 Sampling Signal is -sparse in basis/dictionary WLOG assume sparse in space domain sparse signal nonzero entries

37 Sampling Signal is -sparse in basis/dictionary WLOG assume sparse in space domain Sampling measurements sparse signal nonzero entries

38 Compressive Sampling When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss through linear dimensionality reduction measurements sparse signal nonzero entries

39 How Can It Work? Projection not full rank and so loses information in general Ex: Infinitely many (null space) s map to the same

40 How Can It Work? Projection not full rank and so loses information in general columns But we are only interested in sparse vectors

41 How Can It Work? Projection not full rank and so loses information in general columns But we are only interested in sparse vectors is effectively MxK

42 How Can It Work? Projection not full rank and so loses information in general columns But we are only interested in sparse vectors Design so that each of its MxK submatrices are full rank (ideally close to orthobasis) Restricted Isometry Property (RIP)

43 Restricted Isometry Property (RIP) Preserve the structure of sparse/compressible signals K-dim subspaces

44 Restricted Isometry Property (RIP) Stable embedding RIP of order 2K implies: for all K-sparse x 1 and x 2 K-dim subspaces

45 RIP = Stable Embedding An information preserving projection preserves the geometry of the set of sparse signals RIP ensures that

46 How Can It Work? Projection not full rank and so loses information in general columns Design so that each of its MxK submatrices are full rank (RIP) Unfortunately, a combinatorial, NP-complete design problem

47 Insight from the 70 s [Kashin, Gluskin] Draw at random iid Gaussian iid Bernoulli columns Then provided has the RIP with high probability

48 Randomized Sensing Measurements = random linear combinations of the entries of the signal y = x No information loss for sparse vectors whp measurements sparse signal nonzero entries

49 CS Signal Recovery Goal: Recover signal from measurements Problem: Random projection not full rank (ill-posed inverse problem) Solution: Exploit the sparse/compressible geometry of acquired signal

50 Random projection not full rank CS Signal Recovery Recovery problem: given find Null space Search in null space for the best according to some criterion ex: least squares (N-M)-dim hyperplane at random angle

51 Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Closed-form solution:

52 Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Closed-form solution: Wrong answer!

53 Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Closed-form solution: Wrong answer!

54 Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: find sparsest vector in translated nullspace

55 Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Correct! find sparsest vector in translated nullspace But NP-Complete alg

56 Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Convexify the optimization Candes Romberg Tao Donoho

57 Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Convexify the optimization Correct! Polynomial time alg (linear programming)

58 Compressive Sensing random measurements sparse signal nonzero entries Signal recovery via [Candes, Romberg, Tao; Donoho] optimization

59 Compressive Sensing random measurements sparse signal nonzero entries Signal recovery via iterative greedy algorithm (orthogonal) matching pursuit [Gilbert, Tropp] iterated thresholding [Nowak, Figueiredo; Kingsbury, Reeves; Daubechies, Defrise, De Mol; Blumensath, Davies; ] CoSaMP [Needell and Tropp]

60 Greedy recovery algorithm #1 Consider the following problem min ky xk 2 s.t kxk 0 apple K Suppose we wanted to minimize just the cost, then steepest gradient descent works as bx k = bx k 1 + T (y bx k 1 ) But, the new estimate is no longer K-sparse bx k =thresh bx k 1 + T (y bx k 1 ),K

61 Iterated Hard Thresholding update signal estimate prune signal es4mate (best K- term approx) update residual

62 Greedy recovery algorithm #2 Consider the following problem sparse signal 1 sparse Can we recover the support?

63 Greedy recovery algorithm #2 Consider the following problem sparse signal 1 sparse If then =[ 1, 2,..., N ] arg max h i,yi gives the support of x How to extend to K-sparse signals?

64 Greedy recovery algorithm #2 sparse signal K sparse residue: find atom: r = y bx k 1 k = arg max h i,ri Add atom to support: S = S [ {k} Signal estimate x (Least squares over support) k =( S ) y

65 Orthogonal matching pursuit goal: given y = columns of x, recover a sparse x are unit-norm initialize: bx 0 =0,r = y, ={},i=0 iteration: i = i +1 T r b = k = arg max{ b(1), b(2),..., b(n) } = S k (bx i ) =( ) y, (bx i ) c =0 Find atom with largest support Update signal estimate r = y bx i update residual

66 Specialized solvers CoSAMP [Needell and Tropp, 2009] SPG_l1 [Friedlander, van der Berg, 2008] FPC [Hale, Yin, and Zhang, 2007] AMP [Donoho, Montanari and Maleki, 2010] many many others, see dsp.rice.edu/cs and

67 CS Hallmarks Stable acquisition/recovery process is numerically stable Asymmetrical (most processing at decoder) conventional: smart encoder, dumb decoder CS: dumb encoder, smart decoder Democratic each measurement carries the same amount of information robust to measurement loss and quantization digital fountain property Random measurements encrypted Universal same random projections / hardware can be used for any sparse signal class (generic)

68 Universality Random measurements can be used for signals sparse in any basis

69 Universality Random measurements can be used for signals sparse in any basis

70 Universality Random measurements can be used for signals sparse in any basis sparse coefficient vector nonzero entries

71 Summary: CS Compressive sensing randomized dimensionality reduction exploits signal sparsity information integrates sensing, compression, processing Why it works: with high probability, random projections preserve information in signals with concise geometric structures sparse signals compressible signals

72 Summary: CS Encoding: = random linear combinations of the entries of measurements sparse signal nonzero entries Decoding: Recover from via optimization

73 Image/Video specific signal models and recovery algorithms

74 Transform basis Recall Universality: Random measurements can be used for signals sparse in any basis DCT/FFT/Wavelets Fast transforms; very useful in large scale problems

75 Dictionary learning For many signal classes (ex: videos, light-fields), there are no obvious sparsifying transform basis Can we learn a sparsifying transform instead? GOAL: Given training data x 1,x 2,...,x T xi 2 R N learn a dictionary D, such that s i are sparse. x i = Ds i D 2 R N Q s i 2 R Q

76 Dictionary learning GOAL: Given training data x 1,x 2,...,x T xi 2 R N learn a dictionary D, such that s i are sparse. x i = Ds i D 2 R N Q s i 2 R Q min D,S kx DSk F s.t 8i, ks i k 0 apple K

77 Dictionary learning min D,S kx DSk F s.t 8i, ks i k 0 apple K Non-convex constraint Bilinear in D and S

78 Dictionary learning min D,S kx DSk F + X k ks k k 1 Biconvex in D and S Given D, the optimization problem is convex in s k Given S, the optimization problem is a least squares problem K-SVD: Solve using alternate minimization techniques Start with D = wavelet or DCT bases Additional pruning steps to control size of the dictionary Aharon et al., TSP 2006

79 Dictionary learning min D,S kx DSk F + X k ks k k 1 Pros Ability to handle arbitrary domains Cons Learning dictionaries can be computationally intensive for high-dimensional problems; need for very large amount of data Recovery algorithms may suffer due to lack of fast transforms

80 Models on image gradients Piecewise constant images Sparse image gradients Natural image statistics Heavy tailed distributions

81 Total variation prior TV(I) = X u,v kri(u, v)k 2 = X u,v q I 2 x(u, v)+i 2 y(u, v) TV norm Sparse-gradient promoting norm Formulation of recovery problem min TV(x) s.t y = x

82 Total variation prior Optimization problem Convex Often, works better than transform basis methods Variants 3D (video) Anisotropic TV min TV(x) s.t y = x Code TVAL3 Many many others (see dsp.rice/cs)

83 Beyond sparsity Model-based CS

84 Beyond Sparse Models Sparse signal model captures simplistic primary structure wavelets: natural images Gabor atoms: chirps/tones pixels: background subtracted images

85 Beyond Sparse Models Sparse signal model captures simplistic primary structure Modern compression/processing algorithms capture richer secondary coefficient structure wavelets: natural images Gabor atoms: chirps/tones pixels: background subtracted images

86 Sparse Signals K-sparse signals comprise a particular set of K-dim subspaces

87 Structured-Sparse Signals A K-sparse signal model comprises a particular (reduced) set of K-dim subspaces [Blumensath and Davies] Fewer subspaces <> relaxed RIP <> stable recovery using fewer measurements M

88 Wavelet Sparse Typical of wavelet transforms of natural signals and images (piecewise smooth)

89 Tree-Sparse Model: K-sparse coefficients + significant coefficients lie on a rooted subtree Typical of wavelet transforms of natural signals and images (piecewise smooth)

90 Wavelet Sparse Model: K-sparse coefficients + significant coefficients lie on a rooted subtree RIP: stable embedding K-planes

91 Tree-Sparse Model: K-sparse coefficients + significant coefficients lie on a rooted subtree Tree-RIP: stable embedding [Blumensath and Davies] K-planes

92 Tree-Sparse Model: K-sparse coefficients + significant coefficients lie on a rooted subtree Tree-RIP: [Blumensath and Davies] stable embedding Recovery: inject tree-sparse approx into IHT/CoSaMP

93 Recall: Iterated Thresholding update signal estimate prune signal es4mate (best K- term approx) update residual

94 Iterated Model Thresholding update signal estimate prune signal es4mate (best K- term model approx) update residual

95 Tree-Sparse Signal Recovery signal length N=1024 random measurements M=80 target signal CoSaMP, (RMSE=1.12) Tree-sparse CoSaMP (RMSE=0.037) L1-minimization (RMSE=0.751)

96 Clustered Signals Probabilistic approach via graphical model Model clustering of significant pixels in space domain using Ising Markov Random Field Ising model approximation performed efficiently using graph cuts [Cevher, Duarte, Hegde, Baraniuk 08] target Ising-model recovery CoSaMP recovery LP (FPC) recovery

97 Part 2: Compressive sensing Motivation, theory, recovery Linear inverse problems Sensing visual signals Compressive sensing Theory Hallmark Recovery algorithms Model-based compressive sensing Models specific to visual signals