Modern Signal Processing and Sparse Coding School of Electrical and Computer Engineering Georgia Institute of Technology March 22 2011
Reason D etre Modern signal processing Signals without cosines? Sparse Coding Some applications Theoretical Neuroscience Hyperspectral Images
I require the use of a blade Occam s Razor: A parsimonious shave every time! Given many hypotheses that explain the observations, pick the simplest one!
What s in a definition What is signal processing? Frequency analysis? Filter theory? Applied linear algebra? Signal processing... deals with operations on or analysis of signals... to perform useful operations on those signals. -Wikipedia (you know you d look here first) Methods to extract useful information from measurements of the world around us.
Signals... What is this signal we re processing anyway? Well, what are you interested in? Images Video Music Radar Neurons Inertial measurement units Anything that can be represented by vectors!
...and Systems And all this processing? Again: What interests you? Object classification Motion detection Direction of Arrival Coding Schemes Position determination (The list goes on...)
Two examples: Inverse problems
Classical Fourier Need a principled way to describe signals Classical decomposition: frequency domain F o r i e r
Beyond the Cosine Sine and cosine don t represent all signals well? Example: need a LOT of cosines to represent one discontinuity
Beyond the Cosine 2 An Image example: Original Image FFT: 10K coefs FFT: 25K coefs Wavelet: 5K coefs Wavelet: 10K coefs Wavelet: 25K coefs
Beyond the Cosine 3 Use other shapes: Other orthonormal bases (Gram-Schmidt your favorite functions) Wavelets (localized shapes: still orthonormal) (Tight) frames - more kernels then dimensions!
Sparse Coding Sparse Coding =
Beliefs and Morals Remember Occam s razor? Let s say simple is low dimensional May have a large vector, but in some linear transformation, many of the elements are zero!!
Formalities Signal: y R M can write as y = Φx + ɛ x R N (typically M < N) Overcomplete so choice in x: Choose sparse x
Less Formal Can visualize as: =
Finding Coefficients Your optimal inner product depends on your own system of beliefs and morals. - Dr. J. Romberg Linear (least-squares) version: Pseudo Inverse (or BLUE): ˆx = arg min [ y Φx 2 + λ x 2 2 ] x Optimize l 0 regularized least squares (force sparsity) ˆx = arg min [ y Φx 2 + λ x 2 0 ] x Wonderful result: can optimize l 1 regularized least squares instead ˆx = arg min [ y Φx 2 + λ x 2 1 ] x Now it s convex! (fast solvers)
Aside: Ties to Compressive Sensing y are measurements Φ is a random sensing matrix If x is s-sparse (or s-sparse in some known basis) we can beat the Shannon-Nyquist sampling theorem!
There s an App for that! What we do in the Neurolab: Brains Analog systems Audio Dynamics Manifold Embedding
Brains!! Computational Neuroscience: How can we use mathematical principals (mathematics, engineering, physics) to understand neural systems? The brain takes in sensory information and extracts relevant information Big issue: sensory coding We look at the visual cortex
Visual Pathway Has to interpret natural scenes Many layers (LGN V1 V2,3,4...) (Hubel 1988)
Receptive Fields Encodes via projections onto receptive fields VERY overcomplete set
Non-Linear Effects Inherently linear coding Should be easy enough to test
Surround Suppression Changing field of view changes activity?! (Vinje and Gallant 2002)
Surround Suppression Respons 10 5 0 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 Radius (pixels) (Mengchen and Rozell 2011) Changing field of view changes contrast tuning?! (Alitto and Dan 2010)
Efficient Coding Conclusion: Linear projections are WRONG Natural images have special structure (remember the puppy) Efficient coding hypothesis: Visual Cortex takes advantage of this structure (perhaps using the overcomplete nature of the receptive fields) Can sparse coding explain this? (Barlow 1961)
Optimize for Natural Images What Φ is best for Natural Images? Take example data and optimize: ˆX, ˆΦ = arg min X,Φ [ Y ΦX 2 F + λ k x k 1 ]
Best Kernels? Receptive Fields! Olshausen and Field: optimized and found: (Olshausen and Field 1996)
Analog Systems Need a biologically plausible inference scheme: Locally Competitive Algorithm Find x in an analog system (Rozell et al. 2008)
Analog Systems II Neural implementation Faster than digital - maybe low power
HSI Images Hyperspectral Imagery
HSI Images HSI captures detailed (spectrally) ground images
Unmixing HSI Solve What materials were mixed to create the measurement y = Φx + ɛ Unmixing: optimize coefficients Try sparse priors: ˆx = arg min [ y Φx 2 + λ x 2 1 ] x
Learning Φ Need Φ for the optimization Can use measures spectra in a lab OR: can learn it from the data! ˆX, ˆΦ = arg min X,Φ [ Y ΦX 2 F + λ k x k 1 ]
Learned Spectra And: we recover materials! Northshore Water Sand Alterniflora Reflectance Submerged Net Pine Phragmites Relative Height Wavelength (μm)
Super-Resolution Recover the material coefficients with â = arg min [ y BΦa 2 + λ a 2 1 ] x Then recover the spectrum with ˆx = Φâ
Super-Resolution HSI Resolution Coarse HSI Spectrum Inferred Coefficients Reflectance Summed Reflectance Wavelength (μm) Magnitude Coefficient Number
Classification Results Benefits to multi-class material classification 50-60% faster classification time for same % error Can classify better with smaller training sets Sparse coding is a more general representation!
Closing Remarks Can analyze data in many different ways Simplicity is not a bad thing! Consider what you want to do: then choose the best methods. Signal Processing is not solved - There s a lot of research to do (*nudge**nudge*)
Acknowledgments Thanks to: Dr. Christopher Rozell Dr. Bruno Olshausen Dr. Justin Romberg Han-Lun Yap, Abbie Kressner, Mengchen Zhu
Questions/Comments?