Parameter Estimation in Compressive Sensing. Marco F. Duarte

Parameter Estimation in Compressive Sensing Marco F. Duarte Systems of Lines: Application of Algebraic Combinatorics Worcester Polytechnic Institute, August 13 2015

Concise Signal Structure Sparse signal: only K out of N coefficients are nonzero x = x = n n n model: union of K-dimensional subspaces aligned w/ coordinate axes R N = 1 x sparse signal K nonzero entries

From Samples to Measurements Replace samples by more general encoder based on a few linear projections (inner products), x is sparse y x measurements sparse signal information rate

Example: Line Spectral Estimation Observations: Goal: to estimate the frequencies from y Many existing classes of methods: parametric, subspace, autoregressive Sparsity-based methods can be used when Integrates sparsity/cs with parameter estimation [Chen and Donoho 1998][ ]

Parametric Dictionaries for Sparsity PDs collect observations for a set of values of parameter of interest (one per column) Simple signals (e.g., few complex sinusoids) can be expressed via PDs using sparse coefficient vectors Vector is sparse only in the case when the frequencies in x are among values sampled in the PD Parameter Grid a1 Samples a2 [Gorodntisky and Rao 1997] [Malioutov, Cetin, Willsky 2005] [Cevher, Duarte, Baraniuk 2008] [Cevher, Gurbuz, McClellan, Chellapa 2008][...]

Sparse, but yet not sparse? DFT Basis:

Resolution in Frequency Domain Redundant Fourier Frame, c = 10 Increased resolution allows for more scenes to be formulated as sparse in parametric dictionary Neighboring columns become increasingly similar, thereby increasing dictionary coherence:

Resolution in Frequency Domain Increased resolution allows for more scenes to be formulated as sparse in parametric dictionary Neighboring columns become increasingly similar, thereby increasing dictionary coherence: Theorem: If then for each vector one K-sparse signal there exists at most such that

Greedy Sparse Signal Recovery Iterative Hard Thresholding Inputs: Measurement vector y Sparsity dictionary Sparsity K Output: PD coefficient estimate Initialize: While halting criterion false, (estimate signal) (obtain best sparse approx.) (calculate residual) Return estimate [Blumensath and Davies 2009]

Coherence in Line Spectral Estimation Redundant Fourier Frame, c = 10 Coherence grows as parameter sampling resolution increases: 1 Coherence Dirichlet Kernel

Part 1: Dealing with Coherence Rich Baraniuk (Rice U.)

Issues with Parametric Dictionaries As parameter resolution increases (e.g., larger number of grid points), PD becomes increasingly coherent, hampering sparse approximation algorithms PD s high coherence is a manifestation of resolution issues in underlying estimation problem Structured sparsity models can mitigate this issues by preventing PD elements with coherence above target maximum from appearing simultaneously in recovered signal Norm. Correlation 1 0.8 0.6 0.4 0.2 0 100 200 300 400 Distance between sources, meters Near-Field Localization Norm. Correlation 1 0.8 0.6 0.4 0.2 0 20 40 60 80 Angle between sources, degrees Far-Field Localization [Duarte, 2012]

Structured Sparse Signals A K-sparse signal lives on the collection of K-dim subspaces aligned with coordinate axes R N A K-structured sparse signal lives on a particular (reduced) collection of K-dimensional canonical subspaces R N K K K [Baraniuk, Cevher, Duarte, Hegde 2010]

Leveraging Structure in Recovery Many state-of-the-art sparse recovery algorithms (greedy and optimization solvers) rely on thresholding [Daubechies, Defrise, and DeMol; Nowak, Figueiredo, and Wright; Tropp and Needell; Blumensath and Davies...] s s R N s Thresholding provides the best approximation of s within K K

Structured Recovery Algorithms Modify existing approaches to obtain structure-aware recovery algorithms: replace the thresholding step with a best structured sparse approximation step that finds the closest point within union of subspaces R N s K Greedy structure-aware recovery algorithms inherit guarantees of generic counterparts (even though feasible set may be nonconvex) [Baraniuk, Cevher, Duarte, Hegde 2010]

Structured Frequency-Sparse Signals R N A K-structured PD-sparse signal f consists of K PD elements that are mutually incoherent: if If x is K-structured frequency-sparse, then there exists a K-sparse vector such that and the nonzeros in are spaced apart from each other (band exclusion). [Duarte and Baraniuk, 2012] [Fannjiang and Liao, 2012]

Greedy Sparse Signal Recovery Iterative Hard Thresholding Inputs: Measurement vector y Measurement matrix Sparsity K Output: PD coefficient estimate Initialize: While halting criterion false, (estimate signal) (obtain best sparse approx.) (calculate residual) Return estimate [Blumensath and Davies 2009]

Structured Sparse Signal Recovery Band-Excluding IHT Inputs: Measurement vector y Measurement matrix Structured sparse approx. algorithm Output: PD coefficient estimate Initialize: While halting criterion false, (estimate signal) (obtain band-excluding sparse approx.) Return estimate Can be applied to a variety of greedy algorithms (CoSaMP, OMP, Subspace Pursuit, etc.) (calculate residual) [Duarte and Baraniuk, 2012] [Fannjiang and Liao, 2012]

Median Frequency Estimation Error, Hz Compressive Line Spectral Estimation: 10 2 10 1 10 0 10 1 10 2 Performance Evaluation N = 1024 Standard Sparsity, DFT Basis Standard Sparsity, DFT Frame Structured Sparsity, DFT Frame 100 200 300 400 500 Number of Measurements, M K = 20 c = 5 Df = 0.1 Hz for DFT frame Recovery via IHT Algorithm (or variants) Sufficient separation between adjacent frequencies

Part 2: From Discrete to Continuous Models Hamid Dadkhahi Karsten Fyhn (Aalborg U.) S.H. Jensen (Aalborg U.)

Time Delay of Arrival Estimation: The Basics In radar, we transmit low-autocorrelation signal s(t), (e.g., Alltop seq.), receive reflection from point source at range x and speed v Range and velocity (x,v) of the target can be inferred from time delay-doppler shift

From Discrete to Continuous Models PD can be conceived as a sampling from an infinite set of parametric signals; e.g., TDOA dictionary contains a discrete set of values for the time delay parameter: * Vector varies smoothly in each entry as a function of ; we can represent the signal set as a one-dimensional nonlinear manifold:... T/Nt 2T/Nt 3T/Nt 0 T... s0 Parameter space

From Discrete to Continuous Models For computational reasons, we wish to design methods that allow us to interpolate the manifold from the samples obtained in the PD to increase the resolution of the parameter estimates. An interpolation-based compressive parameter estimation algorithm obtains projection values for sets of manifold samples and interpolates manifold around location of peak projection to get parameter estimate... 0 T/Nt 2T/Nt 3T/Nt... T s0 Parameter space

Interpolating the Manifold: Polar Interpolation s0... All points in manifold have equal norm (delayed versions of fixed waveform) Distance between manifold samples is uniform (depends only on parameter difference, not on parameter values) TDOA manifold features these two properties

Interpolating the Manifold: Polar Interpolation s0... Manifold must be contained within unit Euclidean ball (hypersphere) Manifold has uniform curvature, enabling parameter-independent interpolation scheme Project signal estimates into hypersphere Find closest point in manifold by interpolating from closest samples with polar coordinates

Interpolating the Manifold: Polar Interpolation Find closest point in manifold by interpolating from closest samples with polar coordinates: 0 T Map back from manifold to parameter space to obtain final parameter estimates Akin to Continuous Basis Pursuit (CBP) [Ekanadham, Tranchina, and Simoncelli 2011]

Standard Sparse Signal Recovery Iterative Hard Thresholding Inputs: Measurement vector y Measurement matrix Sparsity K Output: PD coefficient estimate Initialize: While halting criterion false, (estimate signal) (obtain best sparse approx.) (calculate residual) Return estimate [Blumensath and Davies 2009]

Standard Sparse Signal Recovery IHT + Polar Interpolation Inputs: Measurement vector y Measurement matrix Sparsity K Output: PD coefficient estimate Initialize: While halting criterion false, (estimate signal) (approx. signal with polar interp.) Return estimates, (calculate residual) [Fyhn, Dadkhahi, and Duarte 2012][Fyhn, Duarte and Jensen 2015]

Compressive Time Delay Estimation: Performance Evaluation Delay Estimation MSE, μ s 2 10 2 10 4 10 6 10 8 10 10 BOMP BOMP+Poly CS+TDE MUSIC BOMP+CBP BOMP+Polar 0.2 0.4 0.6 0.8 1 CS Subsampling Ratio, M/N Waveform: chirp of length 1 Nt = 500, K = 3, c = 1,. BOMP [Fannjiang and Liao 2012]

Compressive Time Delay Estimation: Performance Evaluation (Noise) Delay Estimation MSE, µs 2 10 0 10 2 10 4 10 6 BOMP BOMP+Poly CS+TDE-MUSIC CS+TED MUSIC BOMP+CBP BOMP+Polar 10 8 0 5 10 15 20 SNR (db) Waveform: chirp of length 1 Nt = 500, K = 3, c = 1,. BOMP [Fannjiang and Liao 2012]

Polar Interpolation for Off-The-Grid Frequency Estimation e(f0-1/c) e(f0) e(f0+1/c) Same properties from time delay estimation manifold also present in the complex exponential manifold: - uniform curvature - equal norm 0 N f Use polar interpolation Map back from manifold to frequency estimates

Compressive Line Spectral Estimation: Mean Parameter Estimation Error (Hz) Performance Evaluation N = 100, K = 4, c = 5, = 0.2 Hz Number of Measurements M BOMP [Fannjiang and Liao 2012] SDP [Tang, Rhaskar, Shah, Recht 2012]

Compressive Line Spectral Estimation: Performance Evaluation (Noise) Mean Parameter Estimation Error (Hz) SNR (db) N = 100, K = 4, M = 50, c = 5, = 0.2 Hz

Compressive Line Spectral Estimation: Computational Expense Noiseless Noisy `1-analysis 9.5245 8.8222 SIHT -synthesis 0.2628 2.9082 0.1499 2.7340 SDP 8.2355 9.9796 BOMP 0.0141 0.0101 CBP 46.9645 40.3477 BISP 5.4265 1.4060 Time (seconds)

Part 3: Meaningful Performance Metrics Dian Mo

Average Parameter Estimation [ µs] Issues with PDs/Structured Sparsity: Sensitivity to Maximal Coherence Value 10 1 10 0 10 1 Parameter T =1 µs T =2 µs T =3 µs T =4 µs T =5 µs 0.2 0.4 0.6 0.8 1 κ Example: Compressive Time Delay Estimation (TDE) with PD and random demodulator Performance depends on measurement ratio Structured sparsity (band exclusion) used to enable highresolution TDE set to optimal value for chirp of length As length of chirp wave increases, performance of compressive TDE varies widely Shape of correlation function dependent on chirp length

Issues with PDs: Euclidean Norm Guarantees Most recovery methods s delay grid provide guarantees to keep Euclidean norm error small This metric, however, is not connected to quality of parameter estimates Example: both estimates have same Euclidean error but provide very different location estimates. We search for a performance metric better suited to the use of PD coefficient vectors (i.e., )

s s Improved Performance Metric: Earth Mover s Distance Earth Mover s Distance (EMD) is based on concept of mass flowing between the entries of first vector in order to match the second EMD value is minimum work needed (measured as mass x transport distance) for first vector to match second: s.t.

s s Improved Performance Metric: Earth Mover s Distance Earth Mover s Distance (EMD) is based on concept of mass flowing between the entries of first vector in order to match the second EMD value is minimum work needed (measured as mass x transport distance) for first vector to match second: When PDs are used, EMD captures parameter estimation error by measuring distance traveled by mass Parameter values must be proportional to indices in PD coefficient vector How to introduce EMD metric into CS recovery process?

s Sparse Approximation with Earth Mover s Distance To integrate into greedy algorithms, we will need to solve the EMD-optimal K-sparse approximation problem s It can be shown that approximation can be obtained by performing K-median clustering on set of points at locations with respective weights Cluster centroids provide support of, values can be easily computed to minimize EMD/estimation error [Indyk and Price 2009]

s Sparse Approximation with Earth Mover s Distance To integrate into greedy algorithms, we will need to solve the EMD-optimal K-sparse approximation problem s One can also show that EMD provides an upper bound for parameter estimation error from PD coefficients: bound tightness depends on dynamic range [Mo and Duarte 2015]

Performance of K-Median Clustering for PD-Based Parameter Estimation Most important factor: autocorrelation function and cumulative autocorrelation function 1 0.8 TDE FE 1 0.8 TDE FE λ(ω) 0.6 0.4 Λ(ω)/Λ( ) 0.6 0.4 0.2 0.2 0 50 0 50 ω/ 0 50 0 50 ω/ [Mo and Duarte 2015] (Note )

Performance of K-Median Clustering for PD-Based Parameter Estimation Minimum separation: Parameter range: Dynamic range: Theorem: Let be sufficiently small and denote the target maximum parameter estimation error by. If and then K-median clustering gives. [Mo and Duarte 2015]

Performance of K-Median Clustering for PD-Based Parameter Estimation Inverse linear relationship between and Λ(σ)/Λ(0) σ [µs] 1.1 0.80 x 10-2 0.60 1.05 0.40 0.20 TDE Λ(σ)/Λ(0) 1.3 1.25 1.2 1.15 FE 0.00 1 0.07 0.7 0.080.72 0.09 0.74 0.1 Λ(ζ)/Λ( ) ζ [µs] 1.1 0.66 0.67 0.68 0.69 Λ(ζ)/Λ( )

Performance of K-Median Clustering for PD-Based Parameter Estimation Inverse linear relationship between and 0.80 x 10-2 0.60 2 TDE 0.20 x 100 0.15 FE σ [µs] 0.40 σ [Hz] 0.10 0.20 0.05 0.00 0.07 0.08 0.09 0.1 ζ [µs] 0.00 0.5 1 1.5 ζ [Hz]

Structured Sparse Signal Recovery Band-Excluding IHT Inputs: Measurement vector y Measurement matrix Structured sparse approx. algorithm Output: PD coefficient estimate Initialize: While halting criterion false, (estimate signal) (obtain band-excluding sparse approx.) Return estimate Can be applied to a variety of greedy algorithms (CoSaMP, OMP, Subspace Pursuit, etc.) (calculate residual) [Duarte and Baraniuk, 2012] [Fannjiang and Liao, 2012]

EMD + Sparse Signal Recovery Clustered IHT Inputs: Measurement vector y Measurement matrix Sparsity K Output: PD coefficient estimate Initialize: While halting criterion false, (estimate signal) (best sparse approx. in EMD) Return estimate (calculate residual) Can be applied to a variety of greedy algorithms (CoSaMP, OMP, Subspace Pursuit, etc.) [Mo and Duarte, 2014]

Average Parameter Estimation [ µs] Mean Parameter Estimation Error 10 1 10 0 10 1 Numerical Results T =1 µs T =2 µs T =3 µs T =4 µs T =5 µs 0.2 0.4 0.6 0.8 1 κ Band-Excluding Subspace Pursuit 0.2 0.4 0.6 0.8 1 κ Example: Compressive TDE w/ PD & random demodulator Performance depends on measurement ratio TDE performance varies widely as chirp length increases Consistent behavior for EMD-based signal recovery, but consistent bias observed Bias partially due to parameter space discretization Average Parameter Estimation [ µs] Mean Parameter Estimation Error 10 1 10 0 10 1 T =1 µs T =2 µs T =3 µs T =4 µs T =5 µs Clustered Subspace Pursuit

Numerical Results Average Parameter Estimation [ µs] Mean Parameter Estimation Error 10 0 10 5 10 10 10 15 10 20 T =1 µs T =2 µs T =3 µs T =4 µs T =5 µs 0.2 0.4 0.6 0.8 1 κ Average Parameter Estimation [ µs] Mean Parameter Estimation Error 10 0 10 5 10 10 10 15 10 20 T =1 µs T =2 µs T =3 µs T =4 µs T =5 µs 0.2 0.4 0.6 0.8 1 κ Band-Excluding Interpolating SP Clustered Interpolating SP Example: Compressive TDE w/ PD & random demodulator Performance depends on measurement ratio When integrated with polar interpolation, performance of compressive TDE improves significantly Sensitivity of Band-Excluding SP becomes more severe, while Clustered SP remains robust

Conclusions In radar and other parameter estimation settings, retrofitting sparsity is not enough! PDs enable use of CS, but often are coherent band exclusion can help, but must be highly precise issues remain with guarantees (Euclidean is not useful) PDs also discretize parameter space, limiting resolution Address discretization with tractable signal models from PDs to manifolds via interpolation techniques readily available models for time delay, frequency/doppler Earth Mover s Distance is a suitable metric easily implementable by leveraging K-median clustering EMD is suitable for dictionaries with well-behaved (compact) correlation functions Ongoing work: multidimensional extensions, sensitivity to noise, theoretical analysis of EMD...

References M. F. Duarte and R. G. Baraniuk, Spectral Compressive Sensing, Applied and Computational Harmonic Analysis, Vol. 35, No. 1, pp. 111-129, 2013. M. F. Duarte, Localization and Bearing Estimation via Structured Sparsity Models, IEEE Statistical Signal Processing Workshop (SSP), Aug. 2012, Ann Arbor, MI, pp. 333-336. K. Fyhn, H. Dadkhahi, and M. F. Duarte, Spectral Compressive Sensing with Polar Interpolation, IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 2013, Vancouver, Canada, pp. 6225-6229. K. Fyhn, M. F. Duarte, and S. H. Jensen, Compressive Parameter Estimation for Sparse Translation-Invariant Signals Using Polar Interpolation, IEEE Transactions on Signal Processing, Vol. 63, No. 4, pp. 870-881, 2015. D. Mo and M. F. Duarte, Compressive Parameter Estimation with Earth Mover s Distance via K-Median Clustering, Wavelets and Sparsity XV, SPIE Optics and Electronics, Aug. 2013, San Diego, CA. D. Mo and M. F. Duarte, Compressive Parameter Estimation via K- Median Clustering, arxiv:1412.6724, December 2014. http://www.ecs.umass.edu/~mduarte mduarte@ecs.umass.edu