Nonlocally Centralized Sparse Representation for Image Restoration

Transcription

1 Nonlocally Centralzed Sparse Representaton for Image Restoraton Wesheng Dong a, Le Zhang b,1, Member, IEEE, Guangmng Sh a, Senor Member, IEEE, and Xn L c, Senor Member, IEEE a Key Laboratory of Intellgent Percepton and Image Understandng (Chnese Mnstry of Educaton), School of Electronc Engneerng, Xdan Unversty, Chna b Dept. of Computng, The Hong Kong Polytechnc Unversty, Hong Kong c Lane Dept. of Comp. Sc. and Elec. Engr., West Vrgna Unversty, Morgantown, WV 6506 Abstract: The sparse representaton models code an mage patch as a lnear combnaton of a few atoms chosen out from an over-complete dctonary, and they have shown promsng results n varous mage restoraton applcatons. However, due to the degradaton of the observed mage (e.g., nosy, blurred and/or downsampled), the sparse representatons by conventonal models may not be accurate enough for a fathful reconstructon of the orgnal mage. To mprove the performance of sparse representaton based mage restoraton, n ths paper the concept of sparse codng nose s ntroduced, and the goal of mage restoraton turns to how to suppress the sparse codng nose. To ths end, we explot the mage nonlocal self-smlarty to obtan good estmates of the sparse codng coeffcents of the orgnal mage, and then centralze the sparse codng coeffcents of the observed mage to those estmates. The so-called nonlocally centralzed sparse representaton (NCSR) model s as smple as the standard sparse representaton model, whle our extensve experments on varous types of mage restoraton problems, ncludng denosng, deblurrng and super-resoluton, valdate the generalty and state-of-the-art performance of the proposed NCSR algorthm. Keywords: Sparse representaton, mage restoraton, nonlocal smlarty 1 Correspondng author: cslzhang@comp.polyu.edu.hk. Ths work s supported by the Hong Kong RGC General Research Fund (PolyU 5375/09E). Ths submsson s an extenson of our paper [] publshed n ICCV

2 I. Introducton Reconstructng a hgh qualty mage from one or several of ts degraded (e.g., nosy, blurred and/or down-sampled) versons has many mportant applcatons, such as medcal magng, remote sensng, survellance and entertanment, etc. For an observed mage y, the problem of mage restoraton (IR) can be generally formulated by y=hx+ υ, (1) where H s a degradaton matrx, x s the orgnal mage vector and υ s the addtve nose vector. Wth dfferent settngs of matrx H, Eq. (1) can represent dfferent IR problems; for example, mage denosng when H s an dentty matrx, mage deblurrng when H s a blurrng operator, mage superresoluton when H s a composte operator of blurrng and down-samplng, and compressve sensng when H s a random projecton matrx [1-3]. In the past decades, extensve studes have been conducted on developng varous IR approaches [4-3]. Due to the ll-posed nature of IR, the regularzaton-based technques have been wdely used by regularzng the soluton spaces [5-9, 1-]. In order for an effectve regularzer, t s of great mportance to fnd and model the approprate pror knowledge of natural mages, and varous mage pror models have been developed [5-8, 14, 17-18, ]. The classc regularzaton models, such as the quadratc Tkhonov regularzaton [8] and the TV regularzaton [5-7] are effectve n removng the nose artfacts but tend to over-smooth the mages due to the pecewse constant assumpton. As an alternatve, n recent years the sparsty-based regularzaton [9-3] has led to promsng results for varous mage restoraton problems [1-3, 16-3]. Mathematcally, the sparse representaton model assumes that a sgnal N x can be represented as x Φα, where Ν M Φ (N<M) s an over-complete dctonary, and most entres of the codng vector α are zero or close to zero. The sparse decomposton of x can be obtaned by solvng an l 0 -mnmzaton problem, formulated as α = arg mn α, st.. x Φα ε, where 0 s a pseudo norm that counts the number of non-zero x α 0 entres n α, and ε s a small constant controllng the approxmaton error. Snce l 0 -mnmzaton s an NP-hard combnatoral optmzaton problem, t s often relaxed to the convex l 1 -mnmzaton. The l 1 -norm based sparse codng problem can be generally formulated n the followng Lagrangan form: α = arg mn { x Φα +λ α }, () x α 1

3 where constant λ denotes the regularzaton parameter. Wth an approprate selecton of the regularzaton parameter λ, we can get a good balance between the sparse approxmaton error of x and the sparsty of α, and the term sparse codng refer to ths sparse approxmaton process of x. Many effcent l 1 -mnmzaton technques have been proposed to solve Eq. (), such as teratve thresholdng algorthms [9-11] and Bregman splt algorthms [5-6]. In addton, compared wth the analytcally desgned dctonares (e.g., wavelet/curvelet dctonary), the dctonares learned from example mage patches can mprove much the sparse representaton performance snce they can better characterze the mage structures [7-8]. Fgure 1: Examples of the sparse codng coeffcents by usng the KSVD [7] approach. The frst row shows some natural mages; the second row shows the correspondng dstrbutons of the sparse codng coeffcents (assocated wth the 3 rd atom of the dctonary n KSVD) of the patches extracted at each pxel. Note that the coeffcents are not randomly dstrbuted but hghly correlated. In the scenaro of IR, what we observed s the degraded mage sgnal y va y=hx+ υ. To recover x from y, frst y s sparsely coded wth respect to Φ by solvng the followng mnmzaton problem: αy = arg mn α{ y HΦα + λ α }, (3) 1 and then x s reconstructed by ˆ x=φα. Clearly, t s expected that α y could be close enough to α x. Due to y the degradaton of the observed mage (e.g., the mage s blurry and nosy), however, t s very challengng to recover the true sparse code α x from y. Usng only the local sparsty constrant α 1 n Eq. (3) may not lead to an accurate enough mage reconstructon. On the other hand, t s known that mage sparse codng coeffcents α are not randomly dstrbuted due to the local and nonlocal correlatons exstng n natural mages. In Fg. 1, we vsualze the sparse codng coeffcents of several example mages. One can see that 3

4 the sparse codng coeffcents are correlated, whle the strong correlatons allow us to develop a much more accurate sparse model by explotng the local and nonlocal redundances. Indeed, some recent works, such as [17] and [18], are based on such consderatons. For example, n [18] a group sparse codng scheme was proposed to code smlar patches smultaneously, and t acheves mpressve denosng results. In ths paper we mprove the sparse representaton performance by proposng a nonlocally centralzed sparse representaton (NCSR) model. To fathfully reconstruct the orgnal mage, the sparse code α y (refer to Eq. (3)) should be as close as possble to the sparse codes α x (refer to Eq. ()) of the orgnal mage. In other words, the dfference υ α = α y α x (called as sparse codng nose, SCN n short, n ths work) should be reduced and hence the qualty of reconstructed mage ˆ x=φαy can be mproved because xˆ x Φα Φα = Φυ. To reduce the SCN, we centralze the sparse codes to some good estmaton of y x α α x. In practce, a good estmaton of α x can be obtaned by explotng the rch amount of nonlocal redundances n the observed mage. The proposed NCSR model can be solved effectvely by conventonal teratve shrnkage algorthm [9], whch allows us to adaptvely adjust the regularzaton parameters from a Bayesan vewpont. The extensve experments conducted on typcal IR problems, ncludng mage denosng, deblurrng and super-resoluton, demonstrate that the proposed NCSR based IR method can acheve hghly compettve performance to state-of-the-art denosng methods (e.g., BM3D [17, 40-4], LSSC [18]), and outperforms state-of-the-art mage deblurrng and super-resoluton methods. The rest of the paper s organzed as follows. Secton II presents the modelng of NCSR. Secton III provdes the teratve shrnkage algorthm for solvng the NCSR model. Secton IV presents extensve expermental results and Secton V concludes the paper. II. Nonlocally centralzed sparse representaton (NCSR) Followng the notaton used n [19], for an mage N x, let x = R x denote an mage patch of sze n n extracted at locaton, where R s the matrx extractng patch x from x at locaton. Gven an dctonary n M Φ, n M, each patch can be sparsely represented as x Φα x, by solvng an 4

5 l 1 -mnmzaton problem α = arg mn { x Φα +λ α }. Then the entre mage x can be x, α 1 represented by the set of sparse codes { α x, }. The patches can be overlapped to suppress the boundary artfacts, and we obtan a redundant patch-based representaton. Reconstructng x from { α x, } s an over-determned system, and a straghtforward least-square soluton s [19]: N T 1 N T x ( R ) ( 1 R 1 x, ) = R Φα. For the convenence of expresson, we let = N T 1 N T ( ) ( 1 1 x, ) = = x Φ α R R R Φα, (4) x = where α x denotes the concatenaton of all α x,. The above equaton s nothng but tellng that the overall mage s reconstructed by averagng each reconstructed patch of x. In the scenaro of mage restoraton (IR), the observed mage s modeled as y=hx+ υ. The sparsty-based IR method recovers x from y by solvng the followng mnmzaton problem: α = arg mn { y HΦ α + λ α }. (5) y α 1 The mage x s then reconstructed as x= ˆ Φ α. y A. The sparse codng nose In order for an effectve IR, the sparse codes α y obtaned by solvng the objectve functon n Eq. (5) are expected to be as close as possble to the true sparse codes α x of the orgnal mage x. However, due to the degradaton of the observed mage y (e.g., nosy and blurred), the sparse code α y wll devate from α x, and the IR qualty depends on the level of the sparse codng nose (SCN), whch s defned as the dfference between α y and α x : υ α = α y α x. (6) To nvestgate the statstcal property of SCN υ α, we perform some experments on typcal IR problems. We use the mage Lena as an example. In the frst experment, we add Gaussan whte nose to the orgnal mage x to get the nosy mage y (the nose level σ n =15). Then we compute α x and α y by solvng Eq. () and Eq. (5), respectvely. The DCT bases are adopted n the experment. Then the SCN υ α s computed. In Fg. (a-1), we plot the dstrbuton of υ α correspondng to the 4 th atom n the dctonary. Smlarly, n Fg. (a-) and Fg. (a-3) we plot the dstrbutons of υ α when the observed data y s blurred (by a Gaussan blur kernel 5

6 wth standard devaton 1.6) and down-sampled by factor 3 n both horzontal and vertcal drectons (after blurred by a Gaussan blur kernel wth standard devaton 1.6), respectvely. We can see that the emprcal dstrbutons of SCN υ α can be well characterzed by Laplacan dstrbutons, whle the Gaussan dstrbutons have much larger fttng errors. To better observe the fttng of the tals, we also plot the these dstrbutons n log doman n Fgs. (b-1)~(b-3). Ths observaton motvates us to model υ α wth a Laplacan pror, as wll be further dscussed n Secton III-A. (a-1) (b-1) (a-) (b-) (a-3) (b-3) Fgure : The dstrbutons of SCN when the Lena mage s (a-1) nosy; (a-) nosy and blurred; and (a-3) down-sampled. (b-1), (b-) and (b-3) show the same dstrbutons of (a-1), (a-) and (a-3) n log doman, respectvely. 6

7 B. Modelng of nonlocally centralzed sparse representaton (NCSR) The defnton of SCN υ α ndcates that by suppressng the SCN υ α we could mprove the IR output ˆx. However, the dffculty les n that the sparse codng vector α x s unknown so that υ α cannot be drectly measured. Nonetheless, f we could have some reasonably good estmaton of α x, denoted by β, avalable, then α y β can be a good estmaton of the SCN υ α. To suppress υ α and mprove the accuracy of α y and thus further mprove the objectve functon of Eq. (5), we can propose the followng centralzed sparse representaton (CSR) model []: αy = arg mn{ y HΦ α + λ α + γ α β }, (7) α 1 p where β s some good estmaton of α, γ s the regularzaton parameter and p can be 1 or. In the above CSR model, whle enforcng the sparsty of codng coeffcents α, the sparse codes are also centralzed to some estmate of α x (.e., β) so that SCN υ α can be suppressed. One mportant ssue of sparsty-based IR s the selecton of dctonary Φ. Conventonal analytcally desgned dctonares, such as DCT, wavelet and curvelet dctonares, are nsuffcent to characterze the so many complex structures of natural mages. The unversal dctonares learned from example mage patches by usng algorthms such as KSVD [7] can better adapt to local mage structures. In general the learned dctonares are requred to be very redundant such that they can represent varous mage local structures. However, t has been shown that sparse codng wth an overcomplete dctonary s unstable [43], especally n the scenaro of mage restoraton. In our prevous work [1], we cluster the tranng patches extracted from a set of example mages nto K clusters, and learn a PCA sub-dctonary for each cluster. Then for a gven patch, one compact PCA sub-dctonary s adaptvely selected to code t, leadng to a more stable and sparser representaton, and consequently better mage restoraton results. In ths paper, we adopt ths adaptve sparse doman selecton strategy but learn the sub-dctonares from the gven mage tself nstead of the example mages. We extract mage patches from mage x and cluster the patches nto K clusters (typcally K=70) by usng the K-means clusterng method. Snce the patches n a cluster are smlar to each other, there s no need to learn an over-complete dctonary for each cluster. Therefore, for each cluster we learn a dctonary of PCA bases and use ths compact PCA dctonary to code the patches n ths cluster. (For the detals of PCA 7

8 dctonary learnng, please refer to [1].) These K PCA sub-dctonares construct a large over-complete dctonary to characterze all the possble local structures of natural mages. In the conventonal sparse representaton models as well as the model n Eq. (7), the local sparsty term α 1 s used to ensure that only a small number of atoms are selected from the over-complete dctonary Φ to represent the nput mage patch. In our algorthm (please refer to Algorthm 1 n Secton III-C), for each patch to be coded, we adaptvely select one sub-dctonary from the traned K PCA sub-dctonares to code t. Ths actually enforces the codng coeffcents of ths patch over the other sub-dctonares to be 0, leadng to a very sparse representaton of the gven patch. In other words, our algorthm wll naturally ensure the sparsty of the codng coeffcents, and thus the local sparsty regularzaton term α 1 can be removed. Hence we propose the followng sparse codng model: αy = arg mn{ y HΦ α + λ α β p}. (8) α There s only one regularzaton term α β n the above model. In the case that p=1, and the estmate p β s obtaned by usng the nonlocal redundancy of natural mages, ths regularzaton term wll become a nonlocally centralzed sparsty term, and we call ths model nonlocally centralzed sparse representaton (NCSR). Next let s dscuss how to obtan a good estmaton β of the unknown sparse codng vectors α. C. The nonlocal estmate of unknown sparse code Generally, there can be varous ways to make an estmate of α x, dependng on how much the pror knowledge of α x we have. If we have many tranng mages that are smlar to the orgnal mage x, we could learn the estmate β of α x from the tranng set. However, n many practcal stuatons the tranng mages are smply not avalable. On the other hand, the strong nonlocal correlaton between the sparse codng coeffcents, as shown n Fg. 1, allows us to learn the estmate β from the nput data. Based on the fact that natural mages often contan repettve structures,.e., the rch amount of nonlocal redundances [31], we search the nonlocal smlar patches to the gven patch n a large wndow centered at pxel. For hgher performance, the search of smlar patches can also be carred out across dfferent scales at the expense of hgher computatonal complexty, as shown n [3]. Then a good estmaton of α,.e., β, can be computed as the weghted average of those sparse codes assocated wth the nonlocal smlar patches (ncludng patch ) 8

9 to patch. For each patch x, we have a set of ts smlar patches, denoted by Ω. Fnally β can be computed from the sparse codes of the patches wthn Ω. Denote by α,q the sparse codes of patch x,q wthn set Ω. Then β can be computed as the weghted average of α,q : β α, (9) = ω q Ω, q, q where ω q, s the weght. Smlar to the nonlocal means approach [31], we set the weghts to be nversely proportonal to the dstance between patches x and x,q : 1 ω ˆ ˆ q, = exp( x x q, / h), (10) W where xˆ = Φα ˆ and xˆ, = Φα ˆ, are the estmates of the patches x and x,q, h s a pre-determned scalar q q and W s the normalzaton factor. In the case of orthogonal dctonares (e.g., the local PCA dctonares T T used n ths work), the sparse codes ˆ α and ˆ α, q can be easly computed as ˆ α ˆ = Φ x and ˆ α ˆ q, = Φ xq,. Our expermental results show that by explotng the nonlocal redundances of natural mages, we are able to acheve good estmaton of the unknown sparse vectors α, and the NCSR model of Eq. (8) can sgnfcantly mprove the performance of the sparsty-based IR results. Eq. (8) can be solved teratvely. We frst ntalze β as 0,.e., β ( 1) =0, and solve for the sparse codng vector, denoted by α (0) y, usng some standard sparse codng algorthm. Then we can get the ntal estmaton of x, denoted by x (0) (0) (0), va x =Φ α. Based on x (0), we search for the smlar patches to each patch, and y hence the nonlocal estmate of β can be updated usng Eqs. (9) and (10). The updated estmaton of α x, denoted by β (0), wll then be used to mprove the accuracy of the sparse codes and thus mprove the IR qualty. Such a procedure s terated untl convergence. In the l th teraton, the sparse vector s obtaned by solvng the followng mnmzaton problem α = arg mn{ y HΦ α + λ α β }. (11) () l () l y p α The restored mage s then updated as () () ˆ l = l y x Φ α. In the above teratve process, the accuracy of sparse () l codng coeffcent α s gradually mproved, whch n turn mproves the accuracy of β. The mproved β y 9

10 are then used to mprove the accuracy of α y, and so on. Fnally, the desred sparse code vector α y s obtaned when the alternatve optmzaton process falls nto a local mnmum. The detaled algorthm wll be presented n Secton III. III. Algorthm of NCSR A. Parameters determnaton In Eq. (8) or Eq. (11) the parameter λ that balances the fdelty term and the centralzed sparsty term should be adaptvely determned for better IR performance. In ths subsecton we provde a Bayesan nterpretaton of the NCSR model, whch also provdes us an explct way to set the regularzaton parameter λ. In the lterature of wavelet denosng, the connecton between Maxmum a Posteror (MAP) estmator and sparse representaton has been establshed, and here we extend the connecton from the local sparsty to nonlocally centralzed sparsty. For the convenence of expresson, let s defne θ=α β. For a gven β, the MAP estmaton of θ can be formulated as θ y = arg max log Ρ ( θ y) θ = arg max {log Ρ( y θ) + log Ρ( θ)}. θ (1) The lkelhood term s characterzed by the Gaussan dstrbuton: 1 1 Ρ( y θ) = Ρ( y αβ, ) = exp( y ) σ HΦ α, (13) πσn n where θ and β are assumed to be ndependent. In the pror probablty Ρ(θ), θ reflects the varaton of α from ts estmaton β. If we take β as a very good estmaton of the sparse codng coeffcent of unknown true sgnal, then θ y =α x β s bascally the SCN assocated wth α y, and we have seen n Fg. that the SCN sgnal can be well characterzed by the Laplacan dstrbuton. Thus, we can assume that θ follows..d. Laplacan dstrbuton, and the jont pror dstrbuton P(θ) can be modeled as 1 θ ( j) Ρ ( θ ) = { exp( )}, (14) σ σ j, j, j where θ (j) are the j th elements of θ, and σ,j s the standard devatons of θ (j). Substtutng Eq. (13) and (14) nto Eq. (1), we obtan 10

11 1 θ arg mn{ Φ α + θ ( j) }. (15) y = y H σ n θ j σ, j Hence, for a gven β the sparse codes α can then be obtaned by mnmzng the followng objectve functon 1 αy = arg mn{ y HΦ α + α ( ) β ( j) }. (16) α σ n j j σ, j Compared wth Eq. (8), we can see that the l 1 -norm (.e., p=1) should be chosen to characterze the SCN term α β. Comparng Eq. (16) wth Eq. (8), we have σ λ n, j=. (17) σ, j In order to have robust estmatons of σ,j, the mage nonlocal redundances can be exploted. In practce, we estmate σ,j usng the set of θ computed from the nonlocal smlar patches. λ,j s then updated wth the updated θ n each teraton or n several teratons to save computatonal cost. Next we present the detaled algorthm of the proposed NCSR scheme. B. Iteratve shrnkage algorthm As dscussed n Secton II, we use an teratve algorthm to solve the NCSR objectve functon n Eq. (8) or (16). In each teraton, for fxed β we solve the followng l 1 -norm mnmzaton problem αy arg mn{ y HΦ α +, j α( j) β ( j) }, (18) = λ α j whch s convex and can be solved effcently. In ths paper we adopt the surrogate algorthm n [9] to solve Eq. (18). In the (l+1)-th teraton, the proposed shrnkage operator for the j th element of α s ( l+ 1) ( l) = τ, j + α ( j) S ( v β ( j)) β ( j), (19) where S τ () s the classc soft-thresholdng operator and () () () v l = K T ( y K α l )/ c + α l, where = K HΦ, K = Φ H T T T, τ=λ,j /c, and c s an auxlary parameter guaranteeng the convexty of the surrogate functon. The dervaton of the above shrnkage operator follows the standard surrogate algorthm n [9]. The nterestng readers may refer to [9] for detals. 11

12 C. Summary of the algorthm As we mentoned n Secton II-B, n our NCSR algorthm the adaptve sparse doman selecton strategy [1] s used to code each patch. We cluster the patches of mage x nto K clusters and learn a PCA sub-dctonary Φ k for each cluster. For a gven patch, we frst check whch cluster t falls nto by calculatng ts dstances to means of the clusters, and then select the PCA sub-dctonary of ths cluster to code t. The proposed NCSR based IR algorthm s summarzed n Algorthm 1. In Algorthm 1, for fxed parameters λ,j and {β } the objectve functon n Eq. (18) s convex and can be effcently solved by the teratve shrnkage algorthm n the nner loop, and ts convergence has been well establshed n [9]. Snce we update the regularzaton parameter λ,j and {β } n every J 0 teratons after solvng a sub-optmzaton problem, Algorthm 1 s emprcally convergent n general, as those presented n [39]. Algorthm 1 1. Intalzaton: (a) Set the ntal estmate as ˆx= y for mage denosng and deblurrng, or ntalze ˆx by bcubc nterpolator for mage super-resoluton; (b) Set ntal regularzaton parameter λ and δ;. Outer loop (dctonary learnng and clusterng): terate on l =1,,, L (a) Update the dctonares {Φ k } va k-means and PCA; (b) Inner loop (clusterng): terate on j=1,,, J ( j+ 1/) ( j) T ( j) (I) xˆ = xˆ + δ H ( y Hx ˆ ), where δ s the pre-determned constant; ( j) ( 1/) ( 1/) (II) Compute [ T ˆ j+,, T ˆ j+ v = Φ ] k R 1 1x Φ k R N Nx, where Φ k s the dctonary assgned to ( 1/) patch ˆ ˆ j + x = Rx ; ( 1) (III) Compute α j+ usng the shrnkage operator gven n Eq. (19); (IV) If mod(j, J 0 )=0 update the parameters λ,j and {β } usng Eqs. (17) and (9), respectvely; ( 1) ( 1) (V) Image estmate update: xˆ j+ = Φ α j+ y usng Eq. (4). IV. Expermental results To verfy the IR performance of the proposed NCSR algorthm we conduct extensve experments on mage denosng, deblurrng and super-resoluton. The basc parameter settng of NCSR s as follows: the patch sze s 7 7 and K=70. The parameters L, J, and δ n Algorthm 1 are set accordngly for dfferent IR applcatons. For mage denosng, δ=0.0, L=3, and J=3; for mage deblurrng and super-resoluton, δ=.4, L=5, and 1

13 J=160. To evaluate the qualty of the restored mages, the PSNR and the recently proposed powerful perceptual qualty metrc FSIM [33] are calculated. Due to the lmted page space, we only show part of the results n ths paper, and all the expermental results can be downloaded on the webste: A. Image denosng Table 1: The PSNR (db) results by dfferent denosng methods. In each cell, the results of the four denosng methods are reported n the followng order: top left SAPCA-BM3D [40]; top rght LSSC [18]; bottom left EPLL [34]; bottom rght NCSR. σ Lena Monarch Barbara Boat C. Man Couple F. Prnt Hll House Man Peppers Straw Average

14 Fgure 3: Denosng performance comparson on the Monarch mage wth moderate nose corrupton. From left to rght and top to bottom: orgnal mage, nosy mage (σ=0), denosed mages by SAPCA-BM3D [40] (PSNR=30.91 db; FSIM=0.9404), LSSC [18] (PSNR=30.58 db; FSIM=0.9310), EPLL [34] (PSNR=30.48 db; FSIM=0.9330), and NCSR (PSNR=30.69 db, FSIM=0.9316). We compare the proposed NCSR method wth three recently developed state-of-the-art denosng methods, ncludng the shape-adaptve PCA based BM3D (SAPCA-BM3D) [40, 41] (whch outperforms the benchmark BM3D algorthm [17]), the learned smultaneously sparse codng (LSSC) method [18] and the expected patch log lkelhood (EPLL) based denosng method [34]. A set of 1 natural mages commonly used n the lterature of mage denosng are used for the comparson study. The PSNR results of the test methods are reported n Table 1 (the hghest PSNR values among the four are hghlghted). Due to the lmted space, the FSIM results are not reported here but they are avalable n the webste of ths work. From Table 1, we can see that the proposed NCSR acheves hghly compettve denosng performance. In term of average PSNR results, NCSR performs almost the same as LSSC, and s slghtly lower than SAPCA-BM3D, whch s the best among the compettors. Let s then focus on the vsual qualty of the denosed mages by the four competng methods. In Fg. 3 and Fg. 4 we show the denosng results on two typcal mages wth moderate nose corrupton and strong nose corrupton, respectvely. It can be seen that NCSR s very effectve n reconstructng both the smooth 14

15 and the texture/edge regons. When the nose level s not very hgh, as shown n Fg. 3 (σ=0), all the four competng methods can acheve very good denosng outputs. When the nose level s hgh, as shown n Fg. 4 (σ=100), however, the SAPCA-BM3D and EPLL methods tend to generate many vsual artfacts. LSSC and NCSR work much better n ths case. In partcular, the denosed mage by the proposed NCSR has much less artfacts than other methods, and s vsually more pleasant. More denosed mages can be downloaded n the webste assocated wth ths paper. Fgure 4: Denosng performance comparson on the House mage wth strong nose corrupton. From left to rght and top to bottom: orgnal mage, nosy mage (σ=100), denosed mages by SAPCA-BM3D [40] (PSNR=5.0 db; FSIM=0.8065), LSSC [18] (PSNR=5.63 db; FSIM=0.8017), EPLL [34] (PSNR=5.44 db; FSIM= ), and NCSR (PSNR=5.65 db; FSIM=0.8068). B. Image deblurrng In ths sub-secton, we conduct experments to verfy the performance of the proposed NCSR method for mage deblurrng n comparson wth some compettve mage deblurrng methods. The deblurrng methods are appled to both the smulated blurred mages and real moton blurred mages. For the smulated blurred mages, the blurred mages are generated by frst applyng a blur kernel and then addng addtve Gaussan nose. Two sets of non-blnd mage deblurrng experments are conducted. Fst, two commonly used blur 15

16 kernels,.e., 9 9 unform blur and D Gaussan functon (non-truncated) wth standard devaton 1.6, are used for smulatons. Addtve Gaussan nose wth nose levels σ n = s added to the blurred mages. Second, 6 typcal non-blnd deblurrng mage experments presented n [37, 4] are conducted to further test the deblurrng performance of the proposed NCSR method under dfferent mage blurry condtons. For the real moton blurred mages, we borrowed the moton blur kernel estmaton method from [35] to estmate the blur kernel and then fed the estmated blur kernel nto the NCSR deblurrng method. For color mages, we only apply the deblurrng operaton to the lumnance component. Table : The PSNR (db) and FSIM results by dfferent deblurrng methods. 9 9 unform blur, σ n = Images Butterfly Boats C. Man House Parrot Lena Barbara Starfsh Peppers Leaves Average FISTA [36] l 0 -SPAR [37] IDD-BM3D [4] ASDS-Reg [1] NCSR FISTA [36] IDD-BM3D [4] ASDS-Reg [1] NCSR Gaussan blur, σ n = We compared the NCSR deblurrng method wth four state-of-the-art deblurrng methods, ncludng the constraned TV deblurrng (denoted by FISTA) method [36], the l 0 -sparsty based deblurrng (denoted by l 0 -SPAR) method [37], the IDD-BM3D deblurrng method [4], and the adaptve sparse doman selecton method (denoted by ASDS-Reg) [1]. Note that FISTA s a TV-based deblurrng approach that can well reconstruct the pecewse smooth regons but often fal to recover fne mage detals. The l 0 -SPAR s a sparsty-based deblurrng method where a fxed sparse doman s used. (Snce l 0 -SPAR does not work well for Gaussan blur kernel, we do not present ts deblurrng results for Gaussan blur kernel n Table ). The recently proposed IDD-BM3D method s an mproved verson of BM3D deblurrng method [0], and 16

17 ASDS-Reg s a very compettve sparsty-based deblurrng method wth adaptve sparse doman selecton. The PSNR and FSIM results on a set of 10 photographc mages are reported n Table. From Table, we can conclude that the proposed NCSR deblurrng method outperforms much the other competng methods. In average NCSR outperforms IDD-BM3D by 0.3 db and 0.17 db for the unform blur and Gaussan blur, respectvely. The NCSR also outperforms ASDS-Reg n average by 0.5 db and 0.98 db for the two dfferent kernels, respectvely. The vsual comparsons of the deblurrng methods are shown n Fgs. 5~6, from whch we can see that the NCSR method produces much cleaner and sharper mage edges and textures than other methods. Table 3 Comparson of the PSNR (db) results of the deblurrng methods. Scenaro Scenaro Method Cameraman (56 56) House (56 56) BSNR Input PSNR TVMM [7] L0-Spar [37] IDD-BM3D[4] NCSR Lena (51 51) Barbara (51 51) BSNR Input PSNR TVMM [7] L0-Spar [37] IDD-BM3D[4] NCSR The PSNR results for the 6 typcal deblurrng experments presented n [37, 4] are reported n Table 3. For far comparson, the PSNR results of other competng methods are drect obtaned from [4]. We optmze the parameters of the proposed deblurrng method for each experment. From Table 3, we can see that both the IDD-BM3D method of [4] and the proposed NCSR method can acheve sgnfcant PSNR mprovement over other competng methods. Parts of the deblurred Barbara mage (scenaro 4) by the competng methods are shown n Fg. 7. We can see that IDD-BM3D and the proposed NCSR method can better recover the fne textures than other competng methods. Moreover, the textures recovered by the proposed NCSR method are better than those by the IDD-BM3D method. 17

18 We also test the proposed NCSR deblurrng method on real moton blurred mages. Snce the blur kernel estmaton s a non-trval task, we borrowed the kernel estmaton method from [35] to estmate the blur kernel and apply the estmated blur kernel n NCSR to restore the orgnal mages. In Fg. 8 we present the deblurrng results of two real blurred mages by the blnd deblurrng method of [35] and the proposed NCSR method. We can see that the mages restored by our approach are much clearer and much more detals are recovered. Consderng that the estmated kernel wll have bas from the true unknown blurrng kernel, these experments valdate that NCSR s robust to the kernel estmaton errors. More moton deblurrng results can be found n the webste of ths paper. Fgure 5: Deblurrng performance comparson on the Cameraman mage. From left to rght and top to bottom: nosy and blurred mage (9 9 unform blur, σ n = ), the deblurred mages by FISTA [36] (PSNR=6.8dB; FSIM=0.867), l 0 -SPAR [37] (PSNR=8.56dB; FSIM= ), IDD-BM3D [4] (PSNR=8.56dB; FSIM=0.9007), ASDS-Reg [1] (PSNR=8.08dB; FSIM=0.8950), and the proposed NCSR (PSNR=8.6dB; FSIM= 0.906). 18

19 Fgure 6: Deblurrng performance comparson on the Starfsh mage. From left to rght and top to bottom: nosy and blurred mage (9 9 unform blur, σ n = ), the deblurred mages by FISTA [36] (PSNR=7.75dB; FSIM=0.8775), l 0 -SPAR [37] (PSNR=8.11dB; FSIM=0.8951), IDD-BM3D [4] (PSNR=9.48dB; FSIM=0.9167), ASDS-Reg [1] (PSNR=9.7 db; FSIM=0.908), and the proposed NCSR (PSNR=30.8dB; FSIM=0.993). Fgure 7: Deblurrng performance comparson on the Barbara (51 51) mage. From left to rght and top to bottom: orgnal mage, nosy and blurred mage (scenaro 4: PSF=[ ] T [ ]/56, σ n =7), the deblurred mages by TVMM [7] (PSNR=4.63dB), l 0 -SPAR [37] (PSNR=4.95dB), IDD-BM3D [4] (PSNR=6.10dB), and the proposed NCSR (PSNR=6.8dB). 19

20 Fgure 8: Deblurrng performance comparson on real moton blurred mages wth the blur kernel estmated usng the kernel estmaton approach from [35]. From left to rght: moton blurred mage, deblurred mage by [35], deblurred mage by proposed NCSR, and close-up vews. C. Image super-resoluton In mage super-resoluton, the low-resoluton (LR) mage s obtaned by frst blurrng the hgh-resoluton (HR) mage wth a blur kernel and then downsamplng by a scalng factor. Hence, recoverng the HR mage from a sngle LR mage s more severely underdetermned than mage deblurrng. In ths subsecton, we test the proposed NCSR based IR for mage super-resoluton. The smulated LR mage s generated by frst blurrng an HR mage wth a 7 7 Gaussan kernel wth standard devaton 1.6, and then downsamplng the blurred mage by a scalng factor 3 n both horzontal and vertcal drectons. The addtve Gaussan noses of standard devaton 5 are also added to the LR mages, makng the IR problem more challengng. Snce human vsual system s more senstve to lumnance changes, we only apply the IR methods to the lumnance component and use the smple bcubc nterpolator for the chromatc components. We compare the proposed NCSR approach wth three recently developed mage super-resoluton methods, ncludng the TV-based method [6], the sparse representaton based method [3], and the ASDS-Reg method [1]. Snce the sparsty-based method n [3] cannot perform the resoluton upscalng We thank the authors of [18], [1], [3], [34-37], [40-4] for provdng ther source codes or expermental results. The code assocated wth ths work wll be avalable onlne. 0

21 and deblurrng smultaneously, as suggested by the authors [3] we apply the teratve back-projecton [38] to the output of method [3] to remove the blur. The PSNR results of the test methods on a set of 9 natural mages are reported n Table 4, from whch we can conclude that the proposed NCSR approach sgnfcantly outperforms the TV [6] and sparsty-based methods [3]. In average the NCSR approach also outperforms the state-of-the-art ASDS-Reg method [1] by up to 0.5 db and 0.3 db for the noseless and nosy cases, respectvely. Ths demonstrates the superorty of the NCSR method for mage nverse problems. The subjectve comparson between the NCSR and other methods are shown n Fgs. 9~11. We can see that the TV-based method [6] tends to generate pecewse constant structures; the mage edges reconstructed by the sparsty-based method [3] contan some vsble artfacts. Obvously, the NCSR approach reconstruct the best vsually pleasant HR mages. The reconstructed edges are much sharper than all the other three competng methods, and more mage fne structures are recovered. Fgure 9: Image super-resoluton performance comparson on Plant mage (scalng factor 3, σ n =0). From left to rght and top to bottom: orgnal mage, LR mage, the reconstructed mages by TV [6] (PSNR=31.34dB; FSIM=0.8909), sparsty-based [3] (PSNR=31.55dB; FSIM=0.8964), ASDS-Reg [1] (PSNR=33.44dB; FSIM=0.94), and the proposed NCSR (PSNR= 34.00dB; FSIM= ). 1

22 Fgure 10: Image super-resoluton performance comparson on Monarch mage (scalng factor 3, σ n =0). From left to rght and top to bottom: orgnal mage, LR mage, the reconstructed mages by TV [6] (PSNR=6.57dB; FSIM=0.8970), sparsty-based [3] (PSNR=4.70dB; FSIM=0.7995), ASDS-Reg [1] (PSNR=7.8dB; FSIM=0.8789) and the proposed NCSR (PSNR= 8.10 db; FSIM=0.9031). Fgure 11: Image super-resoluton performance comparson on Parrot mage (scalng factor 3, σ n =5). From left to rght and top to bottom: orgnal mage, LR mage, the reconstructed mages by TV [6] (PSNR=7.01dB; FSIM=0.856), sparsty-based [3] (PSNR=7.15dB; FSIM=0.863), ASDS-Reg [1] (PSNR=9.01dB; FSIM=0.918), and the proposed NCSR (PSNR= 9.51dB; FSIM= 0.910).

23 Table 4: PSNR (db) and FSIM results (lumnance components) of the reconstructed HR mages. Noseless Images Butterfly flower Grl Pathenon Parrot Raccoon Bke Hat Plants Average TV [6] Sparsty [3] ASDS-Reg [1] NCSR TV [6] Sparsty [3] ASDS-Reg [1] NCSR Nosy V. Concluson In ths paper we presented a novel nonlocally centralzed sparse representaton (NCSR) model for mage restoraton. The sparse codng nose (SCN), whch s defned as the dfference between the sparse code of the degraded mage and the sparse code of the unknown orgnal mage, should be mnmzed to mprove the performance of sparsty-based mage restoraton. To ths end, we proposed a centralzed sparse constrant, whch explots the mage nonlocal redundancy, to reduce the SCN. The Bayesan nterpretaton of the NCSR model was provded and ths endows the NCSR model an teratvely reweghted mplementaton. An effcent teratve shrnkage functon was presented for solvng the l 1 -regularzed NCSR mnmzaton problem. Expermental results on mage denosng, deblurrng and super-resoluton demonstrated that the NCSR approach can acheve hghly compettve performance to other leadng denosng methods, and outperform much other leadng mage deblurrng and super-resoluton methods. References [1] E. Candès and T. Tao, Near optmal sgnal recovery from random projectons: Unversal encodng strateges? IEEE Trans. on Informaton Theory, vol. 5, no. 1, pp , December

24 [] D. Donoho, Compressed sensng, IEEE Trans. on Informaton Theory, vol. 5, no. 4, pp , Aprl 006. [3] E. Candès, J. Romberg, and T. Tao, Robust uncertanty prncples: Exact sgnal reconstructon from hghly ncomplete frequency nformaton, IEEE Trans. on Informaton Theory, vol. 5, no., pp , February 006. [4] M. Bertero and P. Boccacc, Introducton to Inverse Problems n Imagng. Brstol, U.K.: IOP, [5] L. Rudn, S. Osher, and E. Fatem, Nonlnear total varaton based nose removal algorthms, Phys. D, vol. 60, pp , 199. [6] T. Chan, S. Esedoglu, F. Park, and A. Yp, Recent developments n total varaton mage restoraton, n Mathematcal Models of Computer Vson, N. Paragos, Y. Chen, and O. Faugeras, Eds. New York: Sprnger Verlag, 005. [7] J. Olvera, J. M. Boucas-Das, and M. Fgueredo, Adaptve total varaton mage deblurrng: a majorzaton-mnmzaton approach, Sgnal Processng, vol. 89, no. 9, pp , Sep [8] A. N. Tkhonov, Soluton of ncorrectly formulated problems and regularzaton method, In Sovet Math. Dokl., vol. 4, pp , [9] I. Daubeches, M. Defrese, and C. DeMol, An teratve thresholdng algorthm for lnear nverse problems wth a sparsty constrant, Commun. Pure Appl. Math., vol.57, pp , 004. [10] P. Combettes, and V. Wajs, Sgnal recovery by proxmal forward-backward splttng, SIAM J.Multscale Model.Smul., vol.4, pp , 005. [11] M. Zbulevsky and M. Elad, l 1 -l optmzaton n sgnal and mage processng, IEEE Sgnal Processng Magazne, vol. 7, no. 3, pp , May, 010. [1] J. A. Tropp and S. J. Wrght, Computatonal methods for sparse soluton of lnear nverse problems, Proceedngs of IEEE, Specal Issue on Applcatons of Compressve Sensng & Sparse Representaton, vol. 98, no. 6, pp , June, 010. [13] J. M. Boucas-Das, and M.A.T. Fgueredo. A new TwIST: two-step teratve shrnkage/thresholdng algorthms for mage restoraton, IEEE Trans. Image Proc., vol.16, no.1, pp , 007. [14] A. M. Brucksten, D. L. Donoho, and M. Elad, From sparse solutons of systems of equatons to sparse modelng of sgnals and mages, SIAM Revew, vol. 51, no. 1, pp , Feb [15] M. Elad, M.A.T. Fgueredo, and Y. Ma, On the Role of Sparse and Redundant Representatons n Image Processng, Proceedngs of IEEE, Specal Issue on Applcatons of Compressve Sensng & Sparse Representaton, June 010. [16] J. Maral, M. Elad, and G. Sapro, Sparse Representaton for Color Image Restoraton, IEEE Trans. on Image Processng, vol. 17, no. 1, pages 53-69, Jan [17] K. Dabov, A. Fo, V. Katkovnk, and K. Egazaran, Image denosng by sparse 3-D transform-doman collaboratve flterng, IEEE Trans. Image Process., vol. 16, no. 8, pp , Aug [18] J. Maral, F. Bach, J. Ponce, G. Sapro and A. Zsserman, Non-Local Sparse Models for Image Restoraton, n Proc. IEEE Internatonal Conference on Computer Vson, Tokyo, Japan, 009. [19] M. Elad and M. Aharon, Image denosng va sparse and redundant representatons over learned dctonares, IEEE Trans. Image Process., vol. 15, no. 1, pp , Dec [0] K. Dabov, A. Fo, V. Katkovnk, and K. Egazaran, Image restoraton by sparse 3D transform-doman collaboratve flterng, n Socety of Photo-Optcal Instrumentaton Engneers (SPIE) Conference Seres, vol. 681, 008. [1] W. Dong, L. Zhang, G. Sh, and X. Wu, Image deblurrng and super-resoluton by adaptve sparse doman selecton and adaptve regularzaton, IEEE Trans. On Image Processng, vol. 0, no. 7, pp , July 011. [] W. Dong, L. Zhang and G. Sh, Centralzed sparse representaton for mage restoraton, n Proc. IEEE Int. Conf. on Computer Vson (ICCV), 011. [3] Janchao Yang, John Wrght, Thomas Huang and Y Ma, Image Super-Resoluton va Sparse Representaton, IEEE Trans. Image Process., vol. 19, no. 11, pp , Nov [4] S. Chen, D. Donoho, and M. Saunders, Atomc decompostons by bass pursut, SIAM Revew, vol. 43, pp , 001. [5] X. Zhang, M. Burger, X. Bresson, and S. Osher, Bregmanzed nonlocal regularzaton for deconvoluton and sparse reconstructon, SIAM J. Imagng Sc., vol. 3, no. 3, pp , 010. [6] A. Marquna, and S. J. Osher, Image super-resoluton by TV-regularzaton and Bregman teraton, J. Sc. Comput., vol. 37, pp ,

25 [7] M. Aharon, M. Elad, and A. Brucksten, K-SVD: an algorthm for desgnng overcomplete dctonares for sparse representaton, IEEE Trans. Sgnal Process., vol. 54, no. 11, pp , Nov [8] R. Rubnsten, A.M. Brucksten, and M. Elad, Dctonares for sparse representaton modelng, Proceedngs of IEEE, Specal Issue on Applcatons of Compressve Sensng & Sparse Representaton, vol. 98, no. 6, pp , June, 010. [9] L. Sendur and I. W. Selesnck, Bvarate shrnkage functons for wavelet-based denosng explotng nterscale dependency, IEEE Trans. Sgnal Process., vol. 50, no. 11, pp , Nov. 00. [30] I. Ramrez and G. Sapro, Unversal regularzers for robust sparse codng and modelng, arxv: [31] A. Buades, B. Coll, and J. M. Morel, A revew of mage denosng algorthms, wth a new one, Multsc. Model. Smulat., vol. 4, no., pp , 005. [3] D. Glasner, S. Bagon, and M. Iran, Super-resoluton from a sngle mage, IEEE Internatonal Conference on Computer Vson, Tokyo, Japan, 009. [33] L. Zhang, L. Zhang, X. Mou, and D. Zhang, FSIM: A Feature Smlarty Index for Image Qualty Assessment, IEEE Trans. on Image Process., vol. 0, no. 8, pp , Aug [34] D. Zoran and Y. Wess, From learnng models of natural mage patches to whole mage restoraton, n Proc. IEEE Int. Conf. on Computer Vson (ICCV), 011. [35] R. Fergus, B. Sngh, A. Hertzmann, S. T. Rowes, and W. T. Freeman, Removng camera shake from a sngle mage, ACM Trans. Graph. (SIGGRAPH), pages , 006. [36] A. Beck and M. Teboulle, Fast gradent-based algorthms for constraned total varaton mage denosng and deblurrng problems, IEEE Trans. On Image Process., vol. 18, no. 11, pp , Nov [37] J. Portlla, Image restoraton through l 0 analyss-based sparse optmzaton n tght frames, n Proc. IEEE Int. conf. Image Process., pp , Nov [38] M. Iran and S. Peleg, Moton Analyss for Image Enhancement: Resoluton, Occluson, and Transparency, Journal of Vsual Communcaton and Image Representaton, vol. 4, no. 4, pp , Dec [39] E. Candès, M. B. Wakn, and S. P. Boyd, Enhancng sparsty by reweghted l 1 mnmzaton, Journal of Fourer Analyss and Applcatons, vol. 14, pp , 008. [40] V. Katkovnk, A. Fo, K. Egazaran, and J. Astola, From local kernel to nonlocal multple-model mage denosng, Int. J. Computer Vson, vol. 86, no. 1, pp. 1-3, Jan [41] K. Dabov, A. Fo, V. Katkovnk, and K. Egazaran, BM3D mage denosng wth shape-adaptve prncpal component analyss, Proc. Workshop on Sgnal Processng wth Adaptve Sparse Structured Representaton (SPARS 09), Sant-Malo, Fance, Apr [4] A. Danelyan, V. Katkovnk, and K. Egazaran, BM3D frames and varatonal mage deblurrng, IEEE Trans. On Image Processng, vol. 1, no. 4, Apr. 01. [43] M. Elad and I. Yavneh, A pluralty of sparse representaton s better than the sparsest one alone, IEEE Trans. Informaton Theory, vol. 55, no. 10, pp , Oct