Compressive Sensing of High-Dimensional Visual Signals. Aswin C Sankaranarayanan Rice University

Compressive Sensing of High-Dimensional Visual Signals Aswin C Sankaranarayanan Rice University

Interaction of light with objects Reflection Fog Volumetric scattering Human skin Sub-surface scattering Refraction Electron microscopy Tomography

Adelson and Bergen (91) The Plenoptic function Collection of all variations of light in a scene time (1D) space (3D) spectrum (1D) angle (2D) Different slices reveal different scene properties

Plenoptic function time (1D) space (3D) angle (2D) spectrum (1D) High-speed cameras Hyper-spectral imaging Lytro light-field camera

Sensing the plenoptic function Extremely high-dimensional 1000 samples/dim == 10^21 dimensional signal Greater than all the storage in the world Material costs Sensing beyond the visible spectrum Sensing time can be costly Medical imaging Need low-dimensional methods for sensing!

Signal models Many signals exhibit concise geometric structures

Signal models Many signals exhibit concise geometric structures Subspaces Union of subspaces Sparse models Low-dim. non-linear models GOAL: Tractable low-dimensional models for parsimonious sensing and efficient processing of high-dimensional visual data

Roadmap time (1D) space (3D) spectrum (1D) A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging

Linear sensing y = x + e y M measurements x N-dim signal Arbitrary signal Measurement matrix of rank N Number of measurements Least squares recovery M N

Linear sensing y = x + e y M measurements x N-dim signal Is it possible to recover when? No for arbitrary x x M<N

Simple example: 1-sparse signal y = x + e compressive measurements signal Signal prior: 1-sparse signal 1 nonzero entry Possible with M = 2 measurements! y 1 = X k x k = signal value y 2 = X k kx k = signal value location

Candes,(Romberg,(Tao((06),(Donoho((06)( K-sparse signals y = x + e compressive measurements signal Signal recovery K-sparse signal nonzero entries Measurement matrix is i.i.d. sub-gaussian Number of measurements M K log(n/k) Recovery via a convex program

Compressive sensing y = x + e compressive measurements sparse signal Signal recovery K-sparse signal Signal prior Measurement matrix is i.i.d. sub-gaussian nonzero entries Measurement matrix design Number of measurements M K log(n/k) Sampling rate Recovery via a convex program Recovery algorithm

Compressive sensing y = x + e compressive measurements sparse signal System is under-determined nonzero entries Under-sampling factor: M/N Smaller M/N implies gains when sensing is costly

Single-Pixel CS Camera scene single photon detector image reconstruction or processing DMD DMD random pattern on DMD array Kelly lab

Kelly lab, Rice University Single pixel camera Each configuration of micro-mirrors yield ONE compressive measurement Photo-detector A single photodetector tuned to the wavelength of interest Digital micro-mirror device Resolution scalable

First Image Acquisition target 65536 pixels 11000 measurements (16%) 1300 measurements (2%)

Roadmap time (1D) space (3D) spectrum (1D) A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging

SPC on a time-varying scene!me$varying$ scene$ measurements t=1 t=w initial estimate Simple model for recovering video frames from a single-pixel camera (SPC): Group W measurements together to estimate each video frame (image) Value of W ~ aperture time of the video camera

SPC Video Uncertainty Principle t=1 t=w (small) t=w (large) Small%W% Less$mo!on$blur$ More$spa!al$blur$ $ Large%W% More$mo!on$blur$ Less$spa!al$blur$ $ [Wakin, 2010]

SPC Video Uncertainty Principle t=1 t=w (small) t=w (large) total%mse% Find$and$exploit$$ op/mal%balance%% between$spa!al$$ and$mo!on$blurs$ W"

CS-MUVI (CS MUltiscale VIdeo recovery) measurements t=1 t=w t=t 0 t=t 0 +W t=t Step 1: Given knowledge of the speed of the scene s motion, choose W to optimize the spatial/temporal resolution tradeoff total$mse$ W" [AS,$Studer,$Baraniuk$2012]$

CS-MUVI (CS MUltiscale VIdeo recovery) measurements t=1 t=w t=t 0 t=t 0 +W t=t Step 1: Choose W to optimize the spatial/temporal resolution tradeoff Step 2: Acquire full-rate measurements using low-pass codes matched to blur induced by W total$mse$ W"

CS-MUVI (CS MUltiscale VIdeo recovery) measurements t=1 t=w t=t 0 t=t 0 +W t=t low-resolution estimates b 1 b j b F Step 2: Acquire full-rate measurements using low-pass codes matched to blur induced by W Step 3: Recover low-resolution video frames from low-pass measurements (can use simple least squares)

CS-MUVI (CS MUltiscale VIdeo recovery) measurements t=1 t=w t=t 0 t=t 0 +W t=t low-resolution estimates b 1 b j b F Step 3: Recover low-resolution video frames from low-pass measurements (can use simple least squares) Step 4: Estimate the optical flow between low-resolution frames

CS-MUVI (CS MUltiscale VIdeo recovery) measurements t=1 t=w t=t 0 t=t 0 +W t=t low-resolution estimates b 1 b j b F Step 5: Acquire 2 nd set of interleaved measurements using full-resolution codes Step 6: Recover high-resolution video frames from 2 nd set of measurements + low-res video frames, low-res optical flow, sparsity

Multiscale Sensing Codes Problem: 2x measurement rate (low-res and high-res measurement codes) Solution: Design full-res codes that become optimally matched low-res codes when downsampled in space (automatically interleaved) Low-res: Hadamard code High-res: Upsample Hadamard code using random innovations

Multiscale Sensing Codes 1. Start with a row of the Hadamard matrix 2. Upsample$ 3. Add high-freq component Key Idea: Constructing measurement matrices that have good properties when downsampled AS, Studer, Baraniuk (12)

Result

Single pixel camera setup CS-MUVI on SPC

CS-MUVI: IR spectrum InGaAs Photo-detector (Short-wave IR) SPC sampling rate: 10,000 sample/s Number of compressive measurements: M = 16,384 Recovered video: N = 128 x 128 x 61 = 61*M Preview (initial estimate) Upsampled 4x Recovered Video Joint work with Xu, Studer, Kelly, Baraniuk

CS-MUVI on SPC Conventional CS-based recovery CS-MUVI Joint work with Xu, Studer, Kelly, Baraniuk

More results Number of compressive measurements: 65536 Total duration of data acquisition: 6 seconds Reconstructed video resolution: 128x128x256 Preview (6 different videos) *animated gifs* CS-MUVI (6 different videos) *animated gifs*

Achievable Spatial resolution Comparing a recovered frame to a static image of the scene Static Image Full res. Static Image 2x down. Static Image 4x down. CS-MUVI 64x Comp. CS-MUVI Preview

CS-MUVI summary Key points Signal prior: Motion model Measurement matrix: Multiscale design A practical video recovery algorithm for the SPC Scales across wavelengths Extensions to hyper-spectral video camera Coming soon to a camera near you

Roadmap time (1D) space (3D) spectrum (1D) A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging

Hyper-spectral data Hyper-spectral data provides fingerprints for materials Wide range of applications Agriculture, mineralogy Microscopy Surveillance, chemical imaging Hyper-spectral image of the Deepwater Horizon oil spill (Image courtesy SpecTIR.com)

Hyper-spectral data

Signal models for HS data Low rank [x 1,x 2,...,x L ] Rank depends on number of materials in the scene Spectral unmixing problem Sparsity x i be the image at wavelength i Sparse in a wavelet basis

Low rank (subspace models) Individual spectral images lie close to a subspace C x i = C i C 2 R N d i 2 R d Concise geometric structure: d-dim. space d N Subspace spanned by C

Recovery problem Given y i = i x i + e i = i C i + e i Recover Basis subspace coeff. C 1, 2,..., T Challenges: Bilinearity of unknowns Sparsity of C

Recovery problem Given y i = i x i + e i = i C i + e i Structured measurement matrix y i = apple ŷi ỹ i = apple ˆ i C i + e i [ŷ 1, ŷ 2,...ŷ L ]=ˆC [ 1, 2,... L ]+E Singular value decomposition of [ŷ 1, ŷ 2,...ŷ L ]=U V T

Recovery problem Given y i = i x i + e i y i = = i C i + e i Structured measurement matrix apple ŷi ỹ i = apple ˆ i 1. SVD to recover subspace coefficients 2. Solve convex program to recover C C i + e i min kck 1 s.t. 8i, ky i i C ˆ i k 2 apple

CS - Linear dynamical systems Scene x 1 :T Common measurements Innovations measurements Φ Φ ~ t y 1: T y~ 1:T Estimate coefficients αˆ 1:T Estimate matrix C xˆ t = Ĉαˆ Ĉ t xˆ 1:T Measurement matrix is structured Alleviates the bilinearity of unknowns Solving a sequence of linear and convex programs A sparse prior to estimate subspace basis C AS, Turaga, Chellappa and Baraniuk (10)(12)

Hyper-spectral imaging Weather data 2300 Spectral bands in mid-wave to long-wave IR (3.74 15.4 microns) Spatial resolution 128 x 64 Rank 5 Ground Truth 2% 1%%

M/N = 10% M/N = 2% M/N = 1% (rank = 20)

Broader applications Low-rank models are widely applicable Video MRI Ground truth M/N = 2%

Broader applications Low-rank models are widely applicable Traffic video Video MRI Classification on compressive data Purposive sensing Ground truth Recovered video M/N = 4% Expt%1% Expt%2% Expt%3% Expt%4% Oracle$LDS$ 85.71$ 85.93$ 87.5$ 92.06$ CSOLDS$ 84.12$ 87.5$ 89.06$ 85.91$ Classification performance in [%] (CS-LDS at 4% under-sampling) AS(et(al.((10)(

Summary Visual signals are very high-dimensional Interplay between concise signal models and novel imaging architectures Signal models beyond sparsity Rich models from computer vision and image processing Not always easy to bridge the gap Measurement matrices beyond random iid

Collaborators$ Richard$Baraniuk$ Rama$Chellappa$ Kevin$Kelly$ Christoph$Studer$ Lina$Xu$ Yun$Li$ Pavan$Turaga$

References$ Park and Wakin, A multi-scale framework for compressive sensing of videos, PCS 2009 Reddy, Veeraraghavan, Chellappa, P2C2: Programmable pixel compressive cameras for high speed imaging, CVPR 2011 Sankaranarayanan, Studer, Baraniuk, CS-MUVI: Video compressive sensing for spatial multiplexing cameras, ICCP 2012 Sankaranarayanan, Turaga, Chellappa, Baraniuk, Compressive sensing of dynamic scenes, SIIMS (under review)

Initial estimate is low-resolution Low resolution estimate measurements t=1 t=w t=t 0 t=t 0 +W t=t initial estimate Initial estimate is called the preview Fast to compute!!! Linear estimation Fast transform

CS-MUVI (CS MUltiscale VIdeo recovery) measurements t=1 t=w t=t 0 t=t 0 +W t=t initial estimate motion compensation Estimate the optical flow between low-resolution frames Recovery of video at HIGH resolution with sparse models and optical-flow constraints

CS-MUVI (CS MUltiscale VIdeo recovery) measurements t=1 t=w t=t 0 t=t 0 +W t=t initial estimate motion compensation min P F i=1 k x ik 1 s.t ky t t x t kapple OF constraints between x t and x t 1