Theory and Applications of Compressive Sensing

Transcription

1 Purdue Unversty Purdue e-pubs ECE Techncal Reports Electrcal and Computer Engneerng Theory and Applcatons of Compressve Sensng Atul Dvekar Electrcal and Computer Engneerng, Purdue Unversty, Okan Ersoy Follow ths and addtonal works at: Part of the Sgnal Processng Commons Dvekar, Atul and Ersoy, Okan, "Theory and Applcatons of Compressve Sensng" (2010). ECE Techncal Reports. Paper Ths document has been made avalable through Purdue e-pubs, a servce of the Purdue Unversty brares. Please contact epubs@purdue.edu for addtonal nformaton.

2 Theory and Applcatons of Compressve Sensng Atul Dvekar Okan Ersoy TR-ECE December 8, 2010 School of Electrcal and Computer Engneerng 1285 Electrcal Engneerng Buldng Purdue Unversty West afayette, IN

3 v TABE OF CONTENTS IST OF TABES...v IST OF FIGURES...v ABSTRACT...x PUBICATIONS...x 1. INTRODUCTION Overvew Of Compressve Sensng The Incoherence Parameter Example Of Recovery Usng l 1 Mnmzaton SUPERRESOUTION Superresoluton by Compressve Sensng IMAGE FUSION Conventonal Methods for Image Fuson The IHS transform The Brovey transform Fuson by Prncpal Component Analyss Fuson by wavelet methods Image Fuson by Compressve Sensng Algorthm I Algorthm II A faster algorthm usng the Karhunen-oeve bass A Modfed Brovey Transform Algorthm Comparson of Image Fuson Methods Correlaton Coeffcent Spectral Angle Mapper ERGAS CONTENT BASED IMAGE RETRIEVA Content Based Retreval Archtecture Choce of metrc Choce of sparsfyng bass Generatng a query feature vector Expermental Results... 38

4 v 5. RECOVERY BY MUTIPE PARTIA INVERSIONS Improved Estmator by Multple Partal Inversons Improved Orthogonal Matchng Pursut Algorthm Usng Multple Partal Inversons Use as a Drect Estmator Computatonal Complexty Expermental Results Equaton for Nose Improvement n CoSaMP algorthm usng the modfed estmator SUMMARY IST OF REFERENCES VITA...66

5 v IST OF TABES Table Page 3.1: Comparson of Image Fuson Methods : Comparson of Storage Szes for Image Blocks : Fracton of Successful Recoveres for Standard OMP Algorthm wth N= : Fracton of Successful Recoveres for Modfed OMP Algorthm wth N= : Comparson of mean square error of Bass Pursut(BP) wth combned PartEst and IterEst algorthms : Fracton of Successful Recoveres for Standard CoSaMP Algorthm wth N= : Fracton of Successful Recoveres for Modfed CoSaMP Algorthm wth N=

6 v IST OF FIGURES Fgure Page 1.1: eft: Orgnal mage sze 512x512; Rght: Reconstructed mage usng largest magntude coeffcents : Orgnal sgnal wth 40 nonzero entres on left, recovered sgnal on the rght : Orthogonal Matchng Pursut : Superresoluton by Regularzaton: Top eft: Orgnal mage; Top Rght, Bottom eft: Blurred mages wth subpxel shft; Bottom Rght: Reconstructed mage : Superresoluton by Compressve Sensng: Top eft: Orgnal mage, Top Rght: Blurred mage, Bottom: Reconstructed mage : Orgnal ANDSAT Images: Top eft: Panchromatc mage, resoluton 15m; Top Rght, Mddle eft, Rght: Multspectral mages n bands 2,3,4, resoluton 30m; Bottom: Combned multspectral mages : Result of fuson by standard Prncpal Component Analyss Algorthm : Result of fuson by Algorthm I : Result of fuson by Algorthm II (Segmentaton) : Result of fuson by standard Brovey transform : Result of fuson by modfed Brovey transform : Content Based Retreval by compressed sensng : Orgnal ANDSAT spectral band mages ANDSAT spectral band mages reconstructed by l 1 mnmzaton : Reconstructon error n ANDSAT spectral band mages : Retreval of sngle spectral band mage from AVIRIS sensor: eft: Orgnal mage, Mddle: Reconstructon, Rght: Reconstructon Error... 42

7 v Fgure Page 5.1: Multple Partal Inversons Algorthm PartEst : Example of Improved Estmaton(Top to Bottom): Orgnal Sgnal c wth 60 nonzero T entres; Nosy estmate z y; Nose z-c, sample varance 28.8, Improved Estmator; Nose n mproved estmator, sample varance : Modfed Orthogonal Matchng Pursut usng Multple Inversons : Iteratve Algorthm for Improved Estmator : Sngular value dstrbutons for matrces B and B 1024 wth Φ a 200 x 256 matrx drawn from the Unform Sphercal Ensemble, W=160, A= : Sgnal used for comparson of Bass Pursut wth proposed algorthms : Bas n nose terms: Cumulatve sum for, (1) for a 200*256 matrx Φ from the USE wth 800, subsets randomly selected : CoSaMP algorthm : Modfed CoSaMP algorthm... 60

8 x ABSTRACT Dvekar, Atul, Ph.D. Purdue Unversty, December Theory and Applcatons of Compressve Sensng Maor Professor: Okan Ersoy Ths thess develops algorthms and applcatons for compressve sensng, a topc n sgnal processng that allows reconstructon of a sgnal from a lmted number of lnear combnatons of the sgnal. New algorthms are descrbed for common remote sensng problems ncludng superresoluton and fuson of mages. The algorthms show superor results n comparson wth conventonal methods. We descrbe a method that uses compressve sensng to reduce the sze of mage databases used for content based mage retreval. The thess also descrbes an mproved estmator that enhances the performance of Matchng Pursut type algorthms, several varants of whch have been developed for compressve sensng recovery..

9 x PUBICATIONS 1) Image Fuson by Compressve Sensng, Proc. 17 th Internatonal Conference on Geonformatcs, ) Compact Storage of Correlated Data for Content Based Retreval, Proc. Aslomar Conference on Sgnals, Systems and Computers, 2009.

10 1 1. INTRODUCTION Ths thess develops algorthms and applcatons n an emergng topc of sgnal processng called compressve sensng. Compressve sensng developed from questons rased about the effcency of the conventonal sgnal processng ppelne for compresson, codng and recovery of natural sgnals, ncludng audo, stll mages and vdeo. The usual sequence of steps nvolved ncludes the followng. Frst, the analog sgnal s sampled by a sensor such as a camera to obtan a suffcently large number of dgtal samples. Second, the dgtzed samples are transformed nto a sutable doman to compact the energy (and hence the nformaton) nto a relatvely small number of numbers, called coeffcents. The transformaton s chosen to approxmate the optmal Karhunen-oeve transform and results n a representaton of the orgnal sgnal as a lnear sum of a set of bases weghed by the coeffcents. Most of the coeffcents are small n magntude and only a few coeffcents contan a sgnfcant amount of energy. Ths mples that most of the nformaton n the sgnal s concentrated n only a few bases of the sgnal. Thrd, ths sparsty of transform coeffcents s exploted to effcently code the locatons of the few large coeffcents, and the magntudes of these large coeffcents are quantzed and entropy coded. Fnally, the coded representaton s stored and/or transmtted to a decoder, where the codng and transformaton steps are reversed to obtan a good approxmaton of the orgnal set of dgtal samples, whch can be used for D/A converson and presentaton to a vewer, wth a qualty close to that of the orgnal sampled scene.

11 2 Fg. 1.1: eft: Orgnal mage sze 512x512; Rght: Reconstructed mage usng largest magntude coeffcents Ths model s followed by all modern lossy compresson algorthms for audo, stll mages and vdeo, ncludng the JPEG and JPEG2000 standards for stll mages [1,2], the Set Parttonng n Herarchcal Trees(SPIHT) algorthm for stll mage codng [3], and the MPEG and H.264 standards [4], [5] for vdeo compresson. The JPEG standard and MPEG standards use the 8x8 pxel Dscrete Cosne Transform(DCT) to obtan energy compacton and decorrelaton, whle the JPEG2000 standard and the SPIHT coder use a wavelet bass. Even f only a relatvely small number of the largest magntude coeffcents s transmtted to the decoder, and the remanng are assumed to be zero, a good reproducton of the orgnal mage s obtaned when the transform s nverted. Hence t s suffcent to transmt nformaton only about the most sgnfcant coeffcents to the recever. Ths rases the followng queston: If only a few of the transform doman coeffcents are needed for an acceptable reproducton, s t possble to bypass the step of recordng a large number of samples, transformng them, and then throwng away all the nsgnfcant coeffcents? Can one nstead obtan the sgnfcant coeffcents drectly, even f the locatons of the sgnfcant coeffcents are not known a pror? In a seres of recent papers, ths queston was answered n the affrmatve, and lead to an alternate

12 3 model of samplng and sgnal recovery, called compressve sensng. We present an overvew of the basc prncples of compressve sensng. 1.1 Overvew Of Compressve Sensng Consder an underdetermned system Φ where Φ wth, s a N-dmensonal sgnal and s a length vector of measurements equal to lnear combnatons of c. Suppose that has nonzero elements, and we wsh to recover from. One possble technque s to consder every subset Φ of columns drawn from Φ and test whether t fts by least squares leavng no resdue. However ths requres testng of subsets, whch s nfeasble for even moderate values of and. Recent papers [6,7] show that f has nonzero elements wth and the matrx Φ satsfes some addtonal condtons, then can be recovered ether exactly or wth a small approxmaton error. For example, t s shown n [7] that f matrx Φ satsfes a Restrcted Isometry Property(RIP), then mnmzaton can recover the vector. Explctly, the matrx Φ satsfes the RIP wth parameters for f (1 ) c Φ c (1 ) c I 2 2 (1.1) for every sze subset of columns of Φ. If Φ satsfes the RIP wth and, then can be recovered perfectly by solvng mn c such that y c 1 (1.2) If s not exactly sparse, but the components decay rapdly n magntude, then can be approxmately recovered wth a dstorton that s bounded by C c c c c S * 0 2 S 1 (1.3)

13 4 where s a small constant. The lnear program n Equaton (1.2) s a convex optmzaton problem that can be solved effcently by nteror pont methods. However t s dffcult to prove that a matrx Φ satsfes the RIP, and for large sgnals the convex optmzaton can stll be computatonally slow. 1.2 The Incoherence Parameter An alternatve formulaton to Restrcted Isometry has been defned n [8] that lower bounds the number of samples needed for perfect recovery usng an ncoherence parameter µ. Suppose that V (sze n x n) s an orthogonal matrx satsfyng V T V=nI and max V. Select any M rows from V, to gve the M x N matrx Φ as before. If the, sgnal c has m nonzero values that are ±1, and f M C m n and also 2 0 log( / ) M C n for some constants C 0 and C 1, then wth probablty exceedng 1-δ, the log 2 ( / ) 1 sgnal c can be recovered by solvng the same l 1 - mnmzaton mentoned above. 1.3 Example Of Recovery Usng l 1 Mnmzaton We llustrate the results above wth some examples. The RIP property s satsfed wth hgh probablty for Gaussan matrces,.e., matrces wth entres drawn from a Gaussan dstrbuton [7]. We construct a sze 128 x 200 matrx U wth entres drawn from a 0-mean Gaussan dstrbuton wth varance 1/128. Ths makes all, where U denotes the th column of U. E[ U ] 1 for We form a sparse vector c wth 40 nonzero entres drawn from a random dstrbuton. Ths s used to get y Ux, a length 128 szed sample vector. We then use l 1 -mnmzaton as descrbed above to recover the sgnal x. We show the orgnal sgnal and the recovered sgnal n Fg

14 5 Fg. 1.2: Orgnal sgnal wth 40 nonzero entres on left, recovered sgnal on the rght A second approach to ths problem nvolves greedy algorthms such as Orthogonal Matchng Pursut (OMP) [9] and ts varants [10] [11] [12] [13]. In these algorthms, the proecton Φ of the data s used to dentfy a sngle or a few bases that s/are beleved to be n the true sgnal, and then the component of the data that s spanned by all the bases selected so far s removed, leavng behnd a resdue that s orthogonal to the bases selected. The resdue s then used to dentfy more bases usng z T r. The Orthogonal Matchng Pursut algorthm s lsted n Fg Fg. 1.3: Orthogonal Matchng Pursut

15 6 In [9] t s shown that f s S-sparse and Φ s known wth Φ a samplng matrx consstng of zero mean normal random varables wth equal varances, OMP recovers n teratons f,where s a constant. Falure cases are dscussed n [14]. The CoSaMP algorthm [10] provdes error bounds equvalent to l 1 mnmzaton and the speed of the OMP algorthm provded that the RIP constant. Ths mples a relatvely small range of egenvalues ( ) allowed for each column subset of Φ, and verfyng that Φ satsfes the RIP s also computatonally dffcult. In general, determnstc Matchng Pursut(MP) algorthms suffer from an mportant weakness: t s possble to construct sgnals Φ for whch the MP algorthm makes a wrong choce for a bass beleved to be n the orgnal sgnal, removes ths bass from the samples, and then s led astray n makng future choces. The lterature also contans compressve sensng recovery applcatons where the recovery works very well, even though the Φ matrx contans hghly correlated columns whch do not satsfy any reasonable bound on the RIP constants for even small values of. An example s the face recognton work n [15] where a dctonary contans hghly smlar faces and recognton s successfully carred out by mnmzaton. In ths work the class of faces that contans most of the resultant weghts, s returned as the dentfyng soluton. Indeed, Restrcted Isometry s a suffcent, but not necessary, condton for compressve sensng recovery. In ths work, we am to () explot compressve sensng to develop novel algorthms for problems n remote sensng and () contrbute to the theory of compressve sensng by developng recovery algorthms that are superor to exstng work and provde a dfferent understandng of the topc.

16 7 2. SUPERRESOUTION Superresoluton s a common mage processng operaton that attempts to ncrease the resoluton of an mage gven one or more low resoluton mages and/or a pror model. The desred hgh resoluton mage s related to the avalable mage(s) by a forward model that s represented as a matrx Φ. The matrx typcally contans coeffcents of a low pass flter. et x be the vectorzed N x N hgh resoluton mage to be reconstructed and y be the low-resoluton N/2 x N/2 mage or mages. Then we relate the low resoluton and hgh resoluton mages by y x. We may try to obtan the soluton by solvng 2 ˆx= argmn x y-x 2 (2.1) However, the soluton to ths problem s unstable and can vary wldly due to small changes n the data, or nose. Ths occurs because the matrx Φ has very small sngular values. The problem s sad to be ll-posed because the soluton does not vary n a smooth and contnuous way wth the data. A common soluton [16] s to regularze the problem by addng a constrant that reflects a pror knowledge about the doman of mage x. Ths converts the problem nto a well-posed problem wth a unque and stable soluton. Commonly, the smoothness of the mage a property of most natural mages- s used as a constrant. For example, we modfy the soluton to ˆx= argmn y-x Dx x (2.2) Here D s a hgh pass flter such as the aplacan kernel matrx. The second term penalzes the dfferences between neghborng pxels, and λ s the agrange multpler that determnes the relatve sgnfcance of the frst and second terms. An example of

17 8 superresoluton by regularzaton usng the aplacan kernel and two blurred mages wth subpxel offsets s shown n Fg Fg. 2.1: Superresoluton by Regularzaton: Top eft: Orgnal mage; Top Rght, Bottom eft: Blurred mages wth subpxel shft; Bottom Rght: Reconstructed mage

18 9 2.1 Superresoluton by Compressve Sensng We suggest an alternatve method of superresoluton based on compressve sensng. Ths algorthm uses a dctonary D H of 4x4 pxel patches taken from hgh resoluton tranng mages that have the same statstcal propertes as the mage to be reconstructed. Each patch has ts mean subtracted out. For each patch n D H we produce a lowresoluton sample patch by blurrng wth the same operator Φ used n the forward model. The dctonary D of low-resoluton patches s used for l 1 mnmzaton to reconstruct each 4x4 hgh resoluton patch. To ensure contnuty of features n the reconstructed mage, we use overlapped patches wth the left and upper 1-pxel strps of the current patch taken from the already reconstructed left and upper neghbor patches. Ths provdes 7 more samples to add to D. Thus the basc algorthm s From the tranng mages 1) Obtan a 4x4 sze patch dctonary D H (sze 16*K, where K s the number of samples). 2) For each patch n D H construct a sample vector that has four 2x2 pxel means and the same 7 samples as the left and top 1-pxel strps of the hgh resoluton patch. Ths gves the low resoluton sample dctonary D. Ths has sze 11*K. Fnd the means md H and md of D H and D respectvely. Set D (:, k) D (:, k) md and D (:, k) D (:, k) md for each column k. H H H 3) Normalze each column of D to have unt norm. Store the norms n vector n. To reconstruct the hgh resoluton mage: For each 4x4 patch n raster order and 1 pxel overlap wth prevously reconstructed patches, 1) Use low resoluton pxels (2x2) and samples from left and upper reconstructed patches to construct length 11 vector y. (For the top and leftmost rows of patches, we use an estmate of the top and/or left pxel edges usng a standard method such as Brovey). Set y y md.

19 10 2) Solve mn a 1 such that y=d a 3) Normalze a a./ n. The estmate for the hgh resoluton patch s xˆ DHa mdh. The performance of ths algorthm depends on the smlarty of the patches n the dctonary to the actual patterns present. (If the exact pattern s present n the dctonary t wll always be recovered provded that every par of columns of D s lnearly ndependent). For the andsat tranng mages we used, about patches are suffcent to recover hgh resoluton mage wth suffcent vsual qualty. Snce each low resoluton patch s obtaned by a blurrng operaton, t s possble for two hgh resoluton patches to map to a sngle (or neglgbly dfferent) low resoluton vector. In ths case the l 1 -mnmzaton algorthm can pck the wrong sgnal as the hgh resoluton reconstructon. However, provdng the top and left 1-pxel strps seems to be suffcent to obtan good separaton between otherwse smlar low-resoluton patches. If the mean s subtracted from D we get Gaussan statstcs. In [7] t s proven that such a Gaussan matrx almost always has the Restrcted Isometry Property, and hence l 1 mnmzaton wll recover the correct lnear combnaton to reconstruct the hgh resoluton mage. To reduce the computatonal complexty, t s possble to use the Karhunen- oeve transform matrx V of sze 16 by 16 correspondng to the dctonary D H n place of the full dctonary. Ths s obtaned by fndng the covarance matrx of the columns of D H -md H followed by the Sngular Value Decomposton. We obtan D by multplyng V by an 11*16 proecton matrx P, whch captures the means of the 2x2 blocks, and the exact pxel values from the top and left strps. We used ths technque n our mplementaton. An example of superresoluton by compressve sensng s shown n Fg. 2.2.

20 Fg. 2.2: Superresoluton by Compressve Sensng: Top eft: Orgnal mage, Top Rght: Blurred mage, Bottom: Reconstructed mage 11

21 12 3. IMAGE FUSION Image fuson s a technque that combnes mages of a scene from dfferent sensors to dscover knowledge that s not apparent from any sngle mage alone. Image fuson fnds applcatons n analyss of satellte mages, survellance and securty, and medcal magng, where mages from multple modaltes such as MRI, CT and PET may be combned for better vsualzaton and dagnoss. Remote sensng satelltes such as the ANDSAT seres commonly nclude a hgh resoluton panchromatc camera, and a multspectral sensor wth several bands and lower spatal resoluton n each band than the panchromatc camera. For example, the ANDSAT-7 satellte has a panchromatc camera of 15m resoluton and 7 multspectral band sensors wth resoluton 30m. In ths context, the goal of mage fuson s to combne the hgh resoluton panchromatc mage and the lower resoluton multspectral mages to produce an mage n each multspectral band whch s as close as possble to what would be produced by observng the same ground area by a multspectral sensor wth the same resoluton as the panchromatc camera. Thus the fused mage should match the panchromatc mage n spatal resoluton whle preservng the spectral characterstcs of the low resoluton multspectral mages. Common methods for fuson of remote sensed mages nclude the IHS transform [17], the Brovey transform [18], Prncpal Component Analyss(PCA) [19] and wavelet based methods [20,21]. We brefly revew these methods.

22 Conventonal Methods for Image Fuson The IHS transform The IHS (Intensty-Hue-Saturaton) transform frst converts a RGB color mage nto the IHS space whch s correlated to human color percepton. The low resoluton RGB mage s frst nterpolated to the resoluton of the panchromatc mage. Then the pxels at each spatal locaton denoted R, G and B are transformed to the IHS space. The transformaton s gven by I 1/ 3 1/ 3 1/ 3 0 R v 2 / 6 2 / / 6 G 0 v 1 1/ 2 1/ 2 0 B (3.1) The low resoluton ntensty component n the IHS space I 0 s replaced by the hgh resoluton panchromatc mage I new and the transformaton s nverted to gve R 1 1/ 2 1/ 2 new Inew new G 1 1/ 2 1/ 2 v 1 new B v 2 (3.2) The Brovey transform In ths transform, the magntude of the pxel from the panchromatc mage s dvded n proporton to the relatve strengths of the pxel magntudes for each band. For a gven 2x2 pxel block p from the panchromatc mage, let b be the value of the

23 14 correspondng sngle pxel n the low resoluton mage of band. Then the standard Brovey method gves the fuson result for ths 2x2 block n band by b f = b b =1,2,3 p (3.3) Fuson by Prncpal Component Analyss Ths technque utlzes a transformaton of the orgnal spectral mages nto the Karhunen-oeve bass that decorrelates the spectral bands. et X R G be the pxel B vector at locaton and let µ be the mean of X over all spatal locatons of the 1 T multspectral mages. Then the sample covarance matrx s gven byc X X. N The egenvectors of the covarance matrx provde the optmal decorrelatng bass for the spectral bands, whch s called the Karhunen-oeve bass. et the bass be represented by a 3x3 matrx A. Then we obtan P AX, where P are the prncpal components at pxel. The frst prncpal component PC 0 contans the maxmum energy among all the components. Ths s replaced by the panchromatc mage and the transform A s nverted to gve the fused mage Fuson by wavelet methods A wavelet transform carres out a subband decomposton of an mage, separatng low frequency (smooth) and hgh frequency (edge-lke) features. Ths allows the necton of hgh resoluton (sharp) features from the wavelet transform of the hgh resoluton panchromatc mage nto the wavelet transform of the low resoluton multspectral

24 15 mages. The multspectral mage s nterpolated and transformed to the wavelet doman. The panchromatc mage s also transformed to the wavelet doman. The hgh frequency coeffcents from the panchromatc mage are merged (added) nto the hgh frequency subbands of the multspectral mage, and then the transform s nverted to obtan the fused mage. The crtcally sampled wavelet transform s shft-varant, and does not preserve the edges n the fused mage well. To overcome ths problem an oversampled decomposton known as the a trous wavelet transform [22] s used. Ths s a nonorthogonal wavelet decomposton defned by a flter bank {h } and {g =δ -h } where δ denotes the Kronecker delta functon, and represents the allpass operator. Instead of decmaton, the lowpass flter s upsampled by the approprate power of 2. The detal sgnal s gven by the pxel dfference between two successve approxmatons. The fused mages show better edge contnuty features than wth the crtcally sampled wavelet transform [21]. 3.2 Image Fuson by Compressve Sensng We propose three new algorthms that utlze compressve sensng for mage fuson. Compressve sensng requres the system matrx to satsfy the RIP property. Ths may be acheved n two ways : () by explctly constructng a matrx to satsfy RIP, and () by reducng the problem to a matrx that s known to satsfy RIP, such as a Gaussan matrx. We use the latter method. We frst defne a model for the expected hgh resoluton fused mages gven the data. Ths s used to generate a lbrary of canddates for the hgh resoluton fused mages. Correspondng to each canddate n the hgh resoluton lbrary we generate a feature vector n another lbrary. et the lbrary of canddate mages be D H, and the lbrary of feature vectors be D. For the frst method, we utlze the PCA fuson result as a startng pont. Snce the PCA result has good spatal detal but suffers from spectral dstorton, we modfy the spectral propertes of the PCA result n a random manner whle mantanng the spatal features of the PCA results.

25 Algorthm I We use the PCA algorthm to fuse the low resoluton MS bands wth the panchromatc mage. et the fused mages be b 1, b 2 and b 3. We dvde the fused mages nto 16x16 pxel blocks. et b be the th block of the th band, and block n the orgnal low resoluton band correspondng to ths block. For each 16x16 pxel block 1. For each 4x4 block of 2. Create D H and b, s the number of samples. For each sample n N(0,1) to the mean of each 2x2 block. after addng the random number. 3. For each 2x2 block of length 64 vector y and the vector of means. and 4. et md H be the mean of md from each column of b, subtract out the mean value. d be the 8x8 pxel D, matrces of sze 256*S and 64*S respectvely, where S D H, add a random number drawn from D contans the mean value of each 2x2 block d, subtract out the mean value. Call the resultng D H and D H and md be the mean of D, respectvely. D. Subtract 5. et n be a length S vector wth n = D 2. Normalze D D / n. 6. Solve mn c 1 such that y md = Dc 7. Normalze c c / n. md H 8. Set xˆ md D c. Here x ˆ s the fused result correspondng to the H H th block of the th band and s the vector nterpolated to match the dmensons of x ˆ.

26 17 behnd. Note that after md s subtracted out of D, only the Gaussan resdue s left D s then normalzed to have unt-norm columns. Such a Gaussan matrx s known to satsfy RIP wth overwhelmng probablty [7]. Also, the reconstructed block s lkely to be a sparse lnear combnaton of the columns of the results related to RIP to reconstruct the fused mages. D H. Ths ustfes the use of We utlzed the publc-doman software package l 1 -magc to mplement the l 1 mnmzaton algorthm. We found that about 8000 samples was suffcent to produce acceptable results Algorthm II For the second algorthm we segment the panchromatc mage and randomly modfy the mean values of each regon. To mantan spatal detal, we wsh to preserve the dfference between each par of adacent segments. K K For each K*K block at hgh resoluton, let d be the low-resoluton X block 2 2 from low resoluton MS band. 1. Segment the K*K panchromatc mage block to get C regons. 2. Fnd the adacency matrx for the segmentaton map. 3. To create D H and where S s the number of samples: D, matrces of sze 2 K *S and K 4 2 *S respectvely, sample k, et v be the vectorzed panchromatc mage block wth zero mean. For each

27 18 (a) For each par of adacent regons (, ) n the segmentaton map, fnd a random number r from N (0,1). Add r to each pxel of regon n v, and r to each pxel of regon n v. md H and (b) Set (c) et DHk v. Fnd the mean of each 2x2 pxel block n v to gve md H be the mean of md from each column of D H and D H and md be the mean of D, respectvely (d) et mean( d ), and let yd. D k. D. Subtract (e) et n be a length S vector wth n = D 2. Normalze D D / n. (f) Solve mn c 1 such that y md = Dc (g) Normalze c c / n. (h) Set xˆ mdh DHc. Ths s the fused result. We mproved the performance of ths algorthm by segmentng a lnear combnaton of the panchromatc mage and the nterpolated MS mage for each band. et p dn be a blurred verson of the panchromatc mage wth sze K/ 2* K / 2. et < pdn, b > = p b be the correlaton coeffcent. et dn n b be the nterpolated MS band mages. We use 2 2 z = p (1 ) b n for the segmentaton for each band. The ratonale s that f the correlaton s hgh, the panchromatc mage contans vald nformaton for the band and should be weghed heavly, otherwse the low-resoluton mage should be reled on.

28 A faster algorthm usng the Karhunen-oeve bass In ths algorthm, we utlze tranng samples from panchromatc and MS band mages to learn the statstcs of the mages we wsh to generate. We use the Karhunen- oeve bass that optmally sparsfy the sample vectors. Snce the number of bases s much smaller than the number of samples, l 1 mnmzaton takes much less tme. We use the result of the standard PCA algorthm to extract the desrable propertes for the fuson result. To avod confuson, we refer to the standard PCA algorthm as spca. Snce the spca result shows color dstorton, we use the low resoluton MS band mages to specfy the mean of each 2x2 block n the fuson result, and ensure that ths nformaton s not obtaned from the spca result. We assume that the mage statstcs over 8x8 wndows are almost nvarant from the hghest resoluton to the next coarser resoluton. Ths allows us to tran the algorthm wth low resoluton mages and use them to synthesze hgh resoluton mages. 1. Use the spca algorthm to fuse the low resoluton MS bands wth the panchromatc mage. et the fused mages be b 1, b 2 and b Extract 8x8 pxel blocks from correspondng locatons of the three MS bands to generate a 8x8x3 length sample vector. Construct a lbrary wth K samples of tranng data. covarance matrx 3. et md H be the mean of C = 1 D H. Remove t from each sample of D H of sze 192*K D H. Fnd the K T D =1 H DH. Fnd ts egevector matrx V such that C = K VSV T 4. et b,, 1<= <= <= 4, be a 8x8 matrx that has value 1 2 at the 2x2 block bounded by 2 1, 2 and 2 1,2. et B be the matrx of vectorzed b, for 1<= <= <= 4. et W be the matrx defned as

29 20 B B B Here 0 s a 64*48 matrx of 0s. 5. Defne G, a 192*D sze matrx wth elements drawn from a 0-mean Gaussan dstrbuton wth varance 1/192. We chose D= Fnd P = W T G and R = G WP. et Z = [ WR ]. 7. Defne V = T Z V and md = T Z md H. 8. et n be a length 192 vector wth the norm of each column of V. Normalze each column to unt norm. 9. To fuse the th 8x8 block, (a) Concatenate block of each of b 1, b 2 and b 3 to form vector z. (b) et t be the vector of 4x4 pxel blocks from the MS band mages correspondng to block. (c) Defne y [2 t z R] T T T (d) Solve mn c 1 such that y md = Vc (e) Normalze c c / n. (f) Set xˆ mdh DHc. Ths s the fused result. Here W s a matrx of bass vectors correspondng to 2x2 pxel block means from each band. We defne a set of random bass vectors G from a Gaussan dstrbuton and remove any components of these vectors n the subspace spanned by W to obtan R. The

30 21 result of the spca algorthm s proected onto vectors R, and these proectons together wth the low resoluton pxel values provde the data vector y. For each column n V, a vector s generated n the same manner to obtan V, whch s used for l 1 mnmzaton. 3.3 A Modfed Brovey Transform Algorthm In Algorthm II and the KT-based algorthm, we removed the dstorton n the spca result by modfyng the low resoluton spectral values. Ths leads us to propose a modfcaton for the standard Brovey transformaton method for mage fuson that greatly reduces spectral dstorton whle mantanng the computatonal complexty of the standard Brovey transform. For a gven 2x2 pxel block p from the panchromatc mage, let b be the value of the correspondng sngle pxel n the low resoluton mage of band. Then the standard Brovey method gves the fuson result for ths 2x2 block n band as b f = b b =1,2,3 p (3.4) We may wrte p= p d for the 2x2 panchromatc block, wth the 2x2 block mean and p d the resdue after the mean s removed. Then we have b f b = ( pd ) b =1,2,3 (3.5) Snce each pxel n f b s a multple of the correspondng pxel n b, the Spectral Angle Mapper value remans 0. However the rato p d b =1,2,3 changes from each 2x2 block to

31 22 the next,causng heavy spectral dstorton. We propose a smple modfcaton that reduces the spectral dstorton: b f p (3.6) d = b(1 ) b =1,2,3 Ths reduces the varablty n the rato over the mage whle mantanng the smplcty of the Brovey transform. we found that fne detals were more easly dstngushable n the fused result f a multple, 1< < 2, was used along wth p d n the equaton. We chose = Comparson of Image Fuson Methods We provde a comparson of our fuson results usng three commonly used measures: the Spectral Angle Mapper(SAM) value between the low-resoluton MS mage and the fused mage, the Correlaton Coeffcent between the panchromatc mage and the fused MS mages, and the ERGAS measure [23], whch s desgned specfcally to measure fuson performance (a smaller value ndcates better performance). We frst defne the measures. et p be the K x K panchromatc mage, K K b1 d, b2 d, b 3d be the x low resoluton multspectral band mages, b1, b2, b 3 be the 2 2 nterpolated band mages, and b ˆ 1, b ˆ ˆ 2, b 3 be the fused K x K mages.

32 Correlaton Coeffcent The Correlaton coeffcent for band s defned as CC T ( p ) ( ˆ p b ˆ ) b = p bˆ p bˆ (3.7) Here () defnes the mean of the respectve mage. Hgher values of Correlaton Coeffcent ndcate better spatal fdelty wth the panchromatc mage Spectral Angle Mapper et bˆ ˆ v = ˆ b b ˆ and v b = b b be the pxel vectors at locaton n the fused bands, and n the nterpolated bands respectvely. Then we defne the Spectral Angle Mapper(SAM) value as <, ˆ > K 2 1 v v SAM = arccos K 2 =1 v ˆ v (3.8) It measures the average correlaton between spectral vectors n the orgnal and fused band mages. A low value ndcates good spectral fdelty. Ths value s 0 f each v ˆ s a scalar multple of the correspondng v. Ths happens wth the standard Brovey transform algorthm. However, ths does not necessarly lead to good vsual color fdelty snce the constant changes from one 2x2 block to the next.

33 ERGAS ERGAS s a frequently used qualty measure defned n [23]. It stands for erreur relatve globale admensonnelle de synthese whch means relatve dmensonless global error n synthess. It measures propertes that the synthetc (fused) mage should try to acheve: 1. Each hgh resoluton mage b ˆ on beng degraded (blurred) to low resoluton should be as dentcal as possble to the gven low resoluton mage b d. 2. Each hgh resoluton mage b ˆ should be as smlar to the mages that the multspectral sensor would capture f t worked at the hgher resoluton. 3. The set of hgh resoluton mages b ˆ 1, b ˆ ˆ 2, b 3 should be as dentcal as possble to the multspectral set of mages that the correspondng sensor would observe wth the hghest spatal resoluton h. ERGAS s defned as 2 h 1 RMSE ERGAS = 100 (3.9) l N = Here RMSE s the root mean square error between the fused and nterpolated low-resoluton mage of band, s the mean of the band mage and h l s the rato of the sze of low resoluton to hgh resoluton pxels. (It s 1 2 f each low resoluton mage pxel s the mean of a 2x2 block of hgh resoluton pxels). A low value of ERGAS ndcates good fdelty to the data. We compare the results of our algorthms wth those of the PCA and Brovey transform methods.we see that all the compressve sensng algorthms have much better spectral dstorton performance (as measured by SAM and ERGAS) than the standard

34 25 Brovey and PCA methods. The modfed Brovey transform shows much lower spectral dstorton than the standard method. Table 3.1: Comparson of Image Fuson Methods Method SAM (deg) CC ERGAS Standard Brovey Prncpal Components Compressve sensng-i Compressve sensng-seg Compressve Sensng -KT ow Cost Brovey

35 Fg. 3.1: Orgnal ANDSAT Images: Top eft: Panchromatc mage, resoluton 15m; Top Rght, Mddle eft, Rght: Multspectral mages n bands 2,3,4, resoluton 30m; Bottom: Combned multspectral mages 26

36 Fg. 3.2: Result of fuson by standard Prncpal Component Analyss Algorthm 27

37 Fg. 3.3: Result of fuson by Algorthm I 28

38 Fg. 3.4: Result of fuson by Algorthm II (Segmentaton) 29

39 Fg. 3.5: Result of fuson by standard Brovey transform 30

40 Fg. 3.6: Result of fuson by modfed Brovey transform 31

41 32 4. CONTENT BASED IMAGE RETRIEVA Satellte mage databases store vast volumes of mage data acqured from a varety of satellte platforms.these nclude multspectral and hyperspectral sensors that capture a 3D mage cube for each spatal scene.for example, NASA's AVIRIS sensor can produce a data cube of sze 512*512 pxels and 224 bands wth 16 bts per sample gvng a sze of 112 MB for a sngle spatal scene. NOAA satellte mage archves are expected to grow from 300 TB n 2000 to TB n 2015, manly due to hyperspectral data. Manually annotatng mages n such large archves wth nformaton that can be searched by a text based query s mpractcal. Content Based Image Retreval(CBIR) [24,25] s a technque for recovery of mages from an mage database by specfyng non-textual propertes that the mage s expected to have. These propertes are used to generate a query feature vector. Each mage n the database s assocated wth a smlar feature vector. A sutable metrc s used to fnd the feature vector that best matches the query feature vector, and the correspondng mage s returned as the result of the query. CBIR databases have been prevously developed for remote sensng and medcal applcatons. The feature vectors n CANDID [26] contan hstograms from gray levels and n QBIC [27] [28] contan global characterstcs such as color hstograms, shape parameters and texture values. In [28] spato-spectral propertes of multspectral mages are used for retreval from databases of MS/HS mages. A framework for nformaton mnng n mage databases s presented n [29] ntegratng spectral nformaton from a classfer and spatal nformaton based on Gabor wavelet coeffcents. CBIR can provde search capabltes for MS/HS data that are not possble wth textual queres. For example, a geologst may look for a partcular spectral sgnature

42 33 correspondng to a mneral that s embedded n a hyperspectral mage wth a specfc spatal pattern. The geologst may be presented wth canddate spatal patterns and mneral patterns and asked to select those desred. We descrbe a method to reduce the storage needed for each mage record that utlzes compressve sensng and the nformaton stored n the feature vector. We use correlatons (dot product values) of the mages wth spatal patterns as the elements of our feature vectors. For an mage x, the feature vector s obtaned as y= x, where the rows of contan the spatal patterns.the spatal patterns may be obtaned by randomly samplng the database, from centrods obtaned by clusterng mage patches, or even by samplng random dstrbutons. The frst two knds of spatal patterns provde an ntutve characterzaton of the mage beng descrbed: A hgh correlaton value ndcates that the pattern closely matches the mage content. The correlatons can be obtaned at multple scales, allowng for the possblty of searchng for an mage wth coarse scale global patterns and specfc fne scale features n some locatons. Feature vectors obtaned by correlaton drectly ndcate the pattern content of the mage, and are sutable for reconstructon of the mage by compressve sensng. They can also be augmented by parameters such as color content and texture values utlzed n prevous work f desred. The query feature vector can be generated by correlatons of the spatal patterns wth a known exemplar mage or estmated from a user's udgement of the match between the pattern and the desred mage. We consder both possbltes n the sequel.

43 Content Based Retreval Archtecture Fg. 4.1: Content Based Retreval by compressed sensng The schematc for the CBIR system s shown n Fgure (1). We assume that the multspectral data has K bands. Each band s dvded nto B* B spatal blocks and each cube of sze N = K * B* B pxels s vectorzed and stored as a record x. A query vector 2 s generated as descrbed below.the Eucldean dstance q y s used as a metrc to fnd the feature vector y n the database that s closest to the query feature vector q. The vector y s used to obtan a reconstructon * x of the mage x. The recovered sgnal does not exactly match x because the orgnal mage s compressble, rather than exactly * x S-sparse. We store the error x * x along wth the feature vector y n the database. Snce x * x has far less energy than the orgnal mage, t requres very few bts for storage

44 35 compared to the orgnal mage. The magntude of x * x s bounded as descrbed n the frst chapter. We consder several topcs related to the archtecture below Choce of metrc N vectors If the matrx satsfes the Restrcted Isometry Property, multplyng the length x by produces length M proectons y = x that have the same Eucldean dstance relatonshps as the vectors x..e. f x x < x x, then y y < y y. If has entres from a Gaussan dstrbuton wth unt norm columns, the Johnson-ndenstrauss lemma [30] ndcates that the Eucldean norm s preserved by the proecton. In ths case and also when the rows of are patterns randomly taken from the mage database, all the entres of the feature vector carry equal mportance, and we choose a smple Eucldean norm to dentfy the feature vector that s closest to the query. Other norms such as the l 1 metrc are also possble, and ths s a topc for future research Choce of sparsfyng bass We consdered dfferent choces of sparsfyng bases for ths data. The optmal sparsfyng bass (Karhunen-oeve) for the 3D cube s nfeasble to compute unless N s relatvely small. Instead, we fnd the Karhunen-oeve bass along the spectral dmenson and use a standard 2D-DCT or 2D-wavelet bass for the spatal dmensons. To fnd the Karhunen-oeve bass, we fnd the sample covarance matrx 1 C = S T X =1 X for a S sample collecton of S pxel-wse spectral vectors. Then the egenvectors V of C (such that C = V V T ) gve the optmal decorrelatng bass. The tensor product of V wth the 2D spatal transform gves the 3D sparsfyng bass that sparsfes the 3D mage block x. performance. We fnd that the 2D-DCT and 2D-wavelet bases gave approxmately comparable

45 Generatng a query feature vector We propose the followng methods to generate the query feature vector: Usng an exemplar mage vector q= If the user has an mage x and needs to fnd smlar mages, the query feature x can be drectly generated from x Usng correlaton estmates for proecton patterns If spatal patterns from the database are used as the rows of the matrx, a user may estmate the magntudes of the correlatons between the spatal patterns and the desred mage. The spatal patterns can be presented to the user drectly for estmaton of the correlatons. Developng a system based on user based correlaton estmates s a topc of future research. Such a feature vector can be used to reconstruct the orgnal mage usng l 1 mnmzaton wth performance almost as good as a Gaussan matrx, whch was seen n earler work [31].

46 37 Table 4.1: Comparson of Storage Szes for Image Blocks Source Block SPIHT Sze Sze Type (bpp) andsat 64*64*7 5.2 Gaussan andsat 64*64*7 5.2 Gaussan andsat 64*64*7 5.2 Patches andsat 64*64*7 5.2 Patches andsat 128*128*7 5.1 Gaussan andsat 128*128*7 5.1 Gaussan andsat 128*128*7 5.1 Patches andsat 128*128*7 5.1 Patches AVIRIS 64*64* Gaussan AVIRIS 64*64* Gaussan Tx DWT (Haar) 2D DCT DWT (Haar) 2D DCT DWT (Haar) 2D DCT DWT (Haar) 2D DCT DWT (Haar) 2D DCT Fea. Fea. Vec. Res. Coded Vec. and MSE Res. en. Error (bpp) (bpp)

47 Expermental Results We mplemented ths technque wth several datasets from ANDSAT 7 and AVIRIS multspectral data. The ANDSAT-7 mages are 8-bts per pxel wth 7 spectral bands whle AVIRIS data s 16 bts per pxel n 224 bands. We coded 3D mage blocks of dfferent szes as ndcated n the results. We used generated n two ways : Frst, from a Gaussan dstrbuton wth columns normalzed to have unt norm, and second, from 64*64 sze patches selected from the ANDSAT/AVIRIS databases at random and tled to fll each row of. We also used the 2D DWT wth Haar bases and the 2D DCT as the sparsfyng bases. We wsh to compare the sze of the feature vector and the mage block when stored wth or wthout lossy compresson wth the storage needed for the feature vector and the error x * x between the orgnal mage and the 1 l -reconstructon. ossless storage of mage n databases s usually needed for applcatons such as medcal magng, where compresson artfacts are not acceptable. To satsfy the error bound predcted for compressve sensng recovery the length of the feature vector needs to be 2S to 5S where S s the number of large-energy coeffcents n the sgnal. For multspectral data the large-energy coeffcents are about 2-5% of the data sze. Thus we selected the number of feature vectors to be about 15-25% of the sze of each block and rounded each feature vector sample to smulate storage wth 16 bts. The feature vectors were then used wth for l 1 mnmzaton. We used the l 1 - magc software package for reconstructon. For each settng of and transform we averaged the result of 10 reconstructons. The results are shown n Table 4.1. The column SPIHT sze ndcates the storage n bts per pxel needed to compress and store the 3D mage usng a decorrelatng transform for each pxel-wse spectral vector followed by compressed storage of each prncpal component by the SPIHT coder. The column Tx ndcates the type of transform used for reconstructon. Fea. Vec. en. ndcates the number of 16 bt feature vectors stored per block. Res MSE s the sample varance of

48 39 the pxels n the reconstructon error * x x. The next column s the storage n bts per pxel needed for the reconstructon error. We stored the reconstructon error by quantzng the sgnfcant error coeffcents to acheve a Mean Square Error of less than 1.0 relatve to the actual reconstructon error, and codng the quantzed pxels by run length codng. The last column ndcates the storage needed for the feature vector along wth the error n reconstructon. For the ANDSAT-7 data, we show an example of the stored mage cube, the reconstructon usng l 1 mnmzaton and Gaussan matrces, and the reconstructon error n Fg. 4.2, Fg. 4.3, and Fg. 4.4 respectvely. For the AVIRIS data, we show the result of reconstructon for Band 4 n Fg We see that the reconstructon error n each case s noselke and does not contan any sgnfcant obect lke features. Ths ndcates that most of the sgnfcant nformaton from the orgnal mage s already present n the l 1 reconstructon and the error may not be stored f perfect reconstructon s not necessary.

49 Fg. 4.2: Orgnal ANDSAT spectral band mages 40

50 Fg. 4.3 ANDSAT spectral band mages reconstructed by l 1 mnmzaton 41

51 42 Fg. 4.4: Reconstructon error n ANDSAT spectral band mages Fg. 4.5: Retreval of sngle spectral band mage from AVIRIS sensor: eft: Orgnal mage, Mddle: Reconstructon, Rght: Reconstructon Error

52 43 5. RECOVERY BY MUTIPE PARTIA INVERSIONS 5.1 Improved Estmator by Multple Partal Inversons As already mentoned n Chapter 1, there are two algorthmc approaches to compressve sensng recovery. The frst nvolves solvng a lnear program to mnmze the l 1 norm of the sgnal vector c 1 subect to the data constrant y c. The second s the Orthogonal Matchng Pursut (OMP) algorthm [9] and ts varants [10] [11] [13] [12]. OMP s tself an mprovement over the Matchng Pursut algorthm proposed by Mallat and Zhang [32]. In these algorthms, the proecton z= T y of the data s used to dentfy a sngle or a few bases that s/are beleved to be n the true sgnal, and then the component of the data y that s spanned by all the bases selected so far s removed, leavng behnd a resdue r that s orthogonal to the bases selected. The resdue s then used to dentfy more bases usng z= T r. These proectons can be consdered to be crude estmators for the true vector c. We descrbe an algorthm whch mproves upon these estmators. Ths algorthm uses multple estmates of each component of vector c. These estmates are combned to provde a sngle more accurate estmate for each of the components of c. The combned estmate can be used as a stand alone estmator of c, or to mprove the performance of all MP/OMP type algorthms. The prncple s the followng: Select subsets of columns of, each wth W columns. et the th such subset have columns whose ndces are n a set. In the frst teraton, for each subset, we fnd cˆ = ( ) y. Suppose that a column s T 1 T ncluded n a few of the subsets. We estmate c as the mean of the least square

53 44 estmates for c obtaned from all the sets that t s ncluded n. et S = {,1,,2..} denote the set of ndex sets that contan. et be the th element n for,, S. We fnd 1 cˆ = c ( ). In later teratons, we use the resdue r and S S ˆ =1 choose random subsets out of only the remanng unselected columns. The algorthm s presented n Fg Fg. 5.1: Multple Partal Inversons Algorthm PartEst

54 45 We assume that the lnear combnatons are corrupted by sensor nose represented by a vector e so that y = c e. Consder the least square estmate c ˆ for a partcular subset. et denote the set of ndces from {1.. N } not n.e. columns n not ncluded n. We have denotes the cˆ = ( ) T 1 T T 1 T = c ( ) ( c e) y (5.1) kewse, let, denote the set of ndces from {1.. N } not n,. Also, let P T 1 = ( ),, and let, P denote the th column of P., For each c we obtan S 1 cˆ = cˆ ( ), S =1 (5.2) S 1 T T,, S, =1 = c P ( c e) (5.3) In the lstng of the algorthm n Fg. 5.1, S s gven by the fnal value of R for each. Also, matrx X s ntalzed to have 2A columns, whch should be suffcent to accomodate all the estmates obtaned for any partcular c. Alternatvely, X could be extended column by column f desred. The estmator s based on the followng ntuton. Each cˆ ( ) can be treated as a, nosy observaton of c wth nternal nose component ( ) = P T T c.ths,,,,

55 46 self-nose s due to the nonzero dot products between columns of. If we assume the nose components ( ) to be randomly dstrbuted, the sum of nosy observatons n, equaton for c ˆ s an estmator wth decreasng varance as the number of nosy observatons for each coeffcent c ncreases. In realty, we fnd that the estmator s based. As an example, we consder a set of samples y obtaned as lnear proectons y= c, where s a sze M* N matrx wth M = 200 and N = 256, and wth entres drawn from a Gaussan dstrbuton wth 0 mean and varance 1 M. The proectons = T T z y = c ( I) c used n the OMP algorthm contan the nose vector = ( T Ic ). We also obtan estmates for c usng the algorthm presented n Fg. 5.1, wth c contanng S = 60 nonzero components. The locatons of the nonzero components are selected at random, and each s set to 1 wth sgns equally lkely to be postve or negatve. In the multple nversons algorthm, we select W = 160 columnndces randomly n each subset and set A, the average number of estmates per sgnal component to be 50. The results are shown n Fg Fg. 5.2: Example of Improved Estmaton(Top to Bottom): Orgnal Sgnal c wth 60 nonzero T entres; Nosy estmate z y; Nose z-c, sample varance 28.8, Improved Estmator; Nose n mproved estmator, sample varance 9.1