Super-resolution with Nonlocal Regularized Sparse Representation

Transcription

1 Super-resoluton wth Nonlocal Regularzed Sparse Representaton Wesheng Dong a, Guangmng Sh a, Le Zhang b, and Xaoln Wu c a Key Laboratory of Intellgent Percepton and Image Understandng (Chnese Mnstry of Educaton), School of Electronc Engneerng, Xdan Unversty, Chna b Dept. of Computng, The Hong Kong Polytechnc Unversty, Hong Kong c Dept. of Electrcal and Computer Engneerng, McMaster Unversty, Canada ABSTRACT The reconstructon of a hgh resoluton (HR) mage from ts low resoluton (LR) counterpart s a challengng problem. The recently developed sparse representaton (SR) technques provde new solutons to ths nverse problem by ntroducng the l 1 -norm sparsty pror nto the super-resoluton reconstructon process. In ths paper, we present a new SR based mage super-resoluton by optmzng the objectve functon under an adaptve sparse doman and wth the nonlocal regularzaton of the HR mages. The adaptve sparse doman s estmated by applyng prncpal component analyss to the grouped nonlocal smlar mage patches. The proposed objectve functon wth nonlocal regularzaton can be effcently solved by an teratve shrnkage algorthm. The experments on natural mages show that the proposed method can reconstruct HR mages wth sharp edges from degraded LR mages. Keywords: sparse representaton, nonlocal self-smlarty, super-resoluton, teratve shrnkage algorthm 1. INTRODUCTION Image super-resoluton ams to reconstruct a hgh resoluton (HR) mage from ts degraded and down-sampled low resoluton (LR) counterparts. It has wde applcatons n computer vson, mage enhancement, medcal magng and hgh defnton televsons. A typcal mage super-resoluton process conssts of LR mage denosng, nterpolaton and deblurrng, subjectng to a known magng model of the LR mage. The LR magng process can be modeled by: y=dh x+ n (1) where H s a blurrng operator that mmcs the pont spread functon of a camera, D s a down-samplng operator, n s the nose ntroduced n the LR mage generaton, x s the target HR mage, and y s the observed LR mage. The reconstructon of the orgnal HR mage s a typcal ll-posed nverse problem. Ths problem becomes even more underdetermned n the case that only one LR mage s avalable. In ths paper, we concentrate on the mage superresoluton from only a sngle mage. Conventonal lnear nterpolaton based methods, such as blnear, bcubc and cubc splne nterpolators, reconstruct HR mages wth jaggy and zppng artfacts. To mprove the lnear nterpolators, drectonal nterpolators [1]-[3] have been proposed to perform the nterpolaton along the edge drectons. Especally, n [3] Zhang and Wu optmze the nterpolator based on the local covarance of the mage sgnal and acheve state-of-the-art nterpolaton results. However, all these nterpolaton based approaches do not handle the blurrng and noses n the LR mages. Addtonal steps have to be carred out to remove the blurrng and noses n the LR mages. However, the separated denosng and deblurrng steps do not suffcently explot the nformaton hdden n the magng model n Eq. (1). Another classcal super-resoluton approach s teratve back-projecton (IBP) [4], whch s desgned based on the magng model n Eq. (1). However, the IBP process nvolves much uncertanty n recoverng the HR mages, and hence chessboard and zppng artfacts are commonly observed n the reconstructed HR mages by IBP algorthm. Ths manly because IBP tres to fnd the x that could mnmze y DHx ; however, there could be many possble canddates of x that could make y DHx very small. In other words, the soluton space of IBP s too bg so that the resultng soluton may not be the best one. To mprove the performance of the IBP method, mage pror knowledge that take nto account the local mage edge geometres and nonlocal mage redundances has been ncorporated nto the IBP process n [5] [6]. The regularzed IBP technques by blateral flters [5] and nonlocal means based flters [6] can remove many artfacts of the HR mages generated by the orgnal IBP method. However, ther performances are stll not very satsfyng n recoverng fne mage

2 detals and suppressng noses. To reconstruct more vsually pleasng HR mages, more pror knowledge of natural mages should be used to reduce the uncertanty of the HR mages. Typcally, the regularzed super-resoluton can be formulated by solvng the followng mnmzaton problem: arg mn y DHx +λj ( x ) () x where J(x) s a regularzaton term specfyng the pror knowledge of the HR mage and λ s a scalar balancng between the quadratc fdelty term and the regularzaton term. A well-known regularzaton pror s the mnmum pxel ntensty total varatons (TV) [7]. The TV-regularzed approaches favor the mages wth pecewse smooth edge structures n the soluton space of x; however, the TV-based super-resoluton technques cannot recover mage fne detals and often have starcase artfacts. Other mage prors, such as edge smoothness [8] and gradent profle prors [9] have been proposed for mage super-resoluton, yet the resultng edges look unnatural. A recently proposed mage pror knowledge s sparsty of natural mages. It assumes that a natural mage can be sparsely represented n some specfc doman (e.g. wavelet doman, Fourer doman), or t has a sparse expanson over a dctonary of atoms,.e. x=ψα and most of the coeffcents n α are nearly zero. In general, the sparsty constran on α s mplemented by requrng that the l 1 -norm of α s small enough,.e. α 1 <t. The sparse representaton (SR) technques have been successfully appled to a seres of nverse problems, ncludng mage deblurrng [10], denosng [11] and compressve sensng [1]. In [13], the SR of the HR mage over a learned dctonary was proposed to regularze the mage super-resoluton process. As a learnng-based method, ts performance reles on the tranng set, and the learnng of a unversal dctonary for SR s very complcated. In ths paper, we present a new SR based model to reconstruct an HR mage from ts LR counterpart. The contrbutons of the proposed approach are twofold. Frst, we propose to adaptvely estmate the sparse doman of the HR mage patches usng adaptve prncpal component analyss (PCA) va non-local smlar patch groupng. Second, to further enhance the performance of the proposed approach, a nonlocal self-smlarty quadratc constran s also ntroduced to fully explot the nonlocal mage redundances. Snce both the two ponts use the non-local nformaton of the mage, we call the proposed method non-local regularzed SR for mage super-resoluton. In addton, the proposed mnmzaton problem wth non-local regularzatons can be effcently solved by usng a new famly of numercal algorthms, called teratve shrnkage algorthms [14] [15]. The rest of the paper s organzed as follows. Secton presents the proposed algorthm n detal. Secton 3 conducts experments to valdate the performance of the proposed method. Secton 4 concludes the paper.. THE PROPOSED IMAGE SUPER RESOLUTION APPROACH The sparsty pror of natural mages n a specfc transform doman s an effectve constran to refne the soluton space of mage super-resoluton. Wavelet transform, DCT, curvelet and contourlet transforms are commonly used transforms for sparse mage representaton. However, these transforms use a fxed set of bases, whch lack flexbltes n adaptng to varous complex local structures n natural mages. Therefore, dctonary learnng [16] technques have been proposed to learn a unversal over-complete dctonary of atoms so that the mage can be sparsely coded va l 1 -norm mnmzaton. However, for each mage local patch, there are too many rrelatve atoms n the unversal dctonary, whch degrades the effectveness and effcency of sparse representaton (SR) and hence degrades the mage reconstructon performance. It s of hgh demand that we could adaptvely determne the sparse doman of each local patch. In [17] [18], the prncpal component analyss (PCA) technque was used to adaptvely de-correlate the mage local structures for nose removal. For each mage block, a PCA transformaton matrx s locally computed. To ths end, a set of mage blocks that are smlar to the current mage block s grouped and a PCA transform matrx s computed over the tranng dataset. The PCA transformaton matrx actually defnes a type of mage local sparse doman because the mage local patch can be well reconstructed by usng only a few sgnfcant prncpal components. Dfferent from the wavelet transform, DCT, etc, such a PCA transform s sgnal adaptve. In [16-17], state-of-the-art mage denosng results have been obtaned by thresholdng n the adaptve PCA domans. Inspred by the work n [16-17], n ths paper we propose to use the locally adaptve PCA transform as the adaptve sparse doman. On the other hand, the nonlocal self-smlarty pror s used as regularzaton term n the SR based mage reconstructon. Fnally, an objectve functon wll be

3 constructed, whch s an l 1 -norm and l -norm compounded SR mnmzaton problem. Fortunately, an effcent numercal algorthm can be readly obtaned by usng the recently developed teratve shrnkage (IS) technques [14] [15]..1 Adaptve sparse doman determnaton by local PCA transform To adaptvely compute the sparse doman of each local patch, we model a local mage patch as a vector varable and then calculate ts statstcs by usng ts avalable samples. Specfcally, the adaptve PCA transformaton method proposed n [17] [18] s used here to determne the adaptve sparse doman of each local patch. Denote by v x = [ x1, x, L, x ] T m a vector varable contanng all pxel values wthn a w w mage patch. To compute the PCA transformaton matrx of x v, a tranng dataset of t s needed. Denote by x1,0 x1,1 L x1, n 1 x,0 x,1 x, n 1 X L = (3) M M M M xm,0 xm,1 L xm, n 1 a sample matrx of varable x v, where the th row of sample matrx X, denoted by X = x,0 x,1 L x, n 1, s the sample vector of varable x. The sample vector X s centralzed as X = X μ, where μ s the mean of X. Other rows of X can be centralzed analogously and we denote by X the centralzed matrx of X. Wth X, the covarance matrx of X s calculated by 1 T Ω = XX (4) n The goal of PCA s to fnd an orthonormal transformaton matrx P to decorrelate X, Y = PX, so that the covarance matrx of Y s dagonal. Snce Ω s symmetrcal, by apply the SVD (sngular value decomposton) to t, we have T Ω = ΦΛΦ (5) where Φ = φ1 φ L φm s the m m orthonormal egenvector matrx and Λ = dag{ λ1, λ,..., λm} s the dagonal egenvalue matrx wth λ 1 λ... λm. The terms φ 1, φ,..., φ m and λ1, λ,..., λ m are the egenvectors and egenvalues T 1 T of Ω. By settng Ρ = Φ, X can be decorrelated,.e. Y = PX and Λ = YY. An mportant property of PCA s that t n fully de-correlates the orgnal dataset X. Most of the energy of a sgnal wll concentrate on a small subset of the PCA transformed dataset. To apply the block-based adaptve PCA transform for the gven mage, we need a set of tranng sample for each mage patch x v so that the PCA transformaton matrx P can be computed. To ths end, we can collect every possble mage patches n a large wndow centered on x v. However, there can be many dfferent mage patches from the current one n the search area, whch wll deterorate the estmaton of the covarance and hence the PCA matrx. To exclude these rrelatve tranng samples, a block matchng method s used to select the mage patches wth smlar spatal structures. The mage patches whose Eucld dstances to x v are smaller than a predefned threshold are selected: v v x x < τ. By such constrant, nonlocal mage nformaton can be used to compute the PCA transformaton matrxes. The orderng property of PCA bases allows a good reconstructon of the mage by usng only a few prncpal components, whch naturally provde an adaptve sparse doman of each local patch. By allowng the overlappng of the mage patches, pxels n overlapped regon wll be transformed nto dfferent PCA domans. Ths forms a redundant representaton of mage sgnals, whch s very helpful n suppressng noses.. Sparse representaton wth adaptve sparse doman Snce the problem of mage super-resoluton nvolves severe uncertantes, pror knowledge s requred to regularze the soluton. The proposed SR based mage super-resoluton usng adaptve PCA transformatons can be formulated by arg mn y DHx + λ P x v 1 (6) x

4 where x v s the vector of the th mage patch and P s the computed PCA transformaton matrx of x v. λ s a scalar that balances the l 1 -norm and l -norm terms. A problem of (6) s that the computaton of P requres that the mage sgnal x should be estmated frst. However, the estmaton of x wth Eq. (6) requres that P s avalable. Ths s a chcken and egg dlemma. To solve ths problem, we ntally reconstruct x by settng P as a wavelet transformaton matrx. Once x s ntalzed, P can be estmated, and subsequently x can be updated by solvng Eq. (6). Wth the updated x, the PCA transformaton matrxes P can be further updated. These procedures can be terated to alternatvely optmze the reconstructed HR mage and the transformaton matrxes..3 Regularzaton wth non-local self-smlarty constrant The adaptve block-based PCA transformaton can better characterze the varous local mage structures and hence an adaptve sparse representaton can be acheved, whch s very helpful to mprovng the mage reconstructon performance. However, the estmated PCA transformaton matrxes may not be very accurate due to the lmted nformaton of the orgnal HR mage. Ths wll deterorate the performance of the proposed approach. To mprove the qualty of the reconstructed HR mages, more pror nformaton should be ncorporated. One mportant mage pror s that natural mages often contan repettve patterns and structures throughout the mage. Such non-local statstcal redundances can be very useful n enhancng the qualty of reconstructed mages. Actually, n the PCA-based adaptve sparse doman determnaton process, the non-local redundancy nformaton has been already used for tranng dataset constructon. Inspred by the success of the non-local means flterng for mage denosng [19], we further ntroduce a nonlocal self-smlarty quadratc constrant nto the super-resoluton process to fully explot the nonlocal redundances. Wth ths quadratc constrant as another regularzaton term, the mage superresoluton model n Eq. (6) s lfted to: arg mn x v v v y DHx + η x γ x + λ P x j j 1 j where η s a scalar parameter to balance the non-local regularzaton term; x v j s the j th smlar (vectorzed) block to x v n a nonlocal neghborhood; γ j s the weght assgned to x v j. We use the block matchng method to locate the smlar blocks to x v v v n a large enough wndow: x xj < T, where T s a predefned threshold. The weghts γ j depend on the smlarty between x v and x v j, whch can be calculated by: v v x xj h 1 γ j = e (8) c where h s a scalar controllng the smlarty and c s the normalzaton factor. As n secton., n Eq. (7) the selecton of smlar nonlocal neghbors and the computaton of the weghts requre that an ntal estmate of the orgnal HR mage, and then Eq. (7) can be teratvely solved. (7).4 Numercal algorthm Eq. (7) s a mnmzaton problem wth compound regularzaton terms. For the convenence of expresson, we can rewrte Eq. (7) n a matrx form as follows arg mn y DHx + η ( I A) x + λ Px (9) x 1 where P s form by all the P ; A s a matrx of dmenson of N N wth N beng the dmenson of the target HR mage. A s set as follows vm th v v vm γ j, f xn s the j smlar neghbor to xm, and j are the locatons of xm and xn Amn (, ) = (10) 0, otherwse Eq. (9) can be expressed as

5 Let arg mn X y DH + λ P η( I A) x x 0 1 (11) Snce P s orthogonal, we have y y% = 0, DH K = η( I A), u = Px (1) T x= P u. Then (11) can be rewrtten as follows arg mn u y% u λ u (13) T KP + Eq. (13) s a challengng large-scale compound l 1 -norm and l -norm mnmzaton problem. Conventonal optmzaton technques, such as steepest-descent, conjugate gradent and nteror-pont algorthms, are neffcent n solvng ths mnmzaton problem. In ths paper, we adopt the recently proposed teratve shrnkage algorthm [14] [15] to solve ths mnmzaton problem. We summarze the teratve algorthm for solvng (13) n Algorthm 1. 1 Algorthm 1 for solvng Eq. (13) 1 Intalzaton: ntalze x by settng P as the wavelet transformaton matrx; then A can be ntalzed, and set (0) u = 0. Iterate on k untl convergence ( k+ 1/) ( k) T T ( k) (a) u = u + K ( y% KP u ); ( k+ 1) ( k+ 1/) (b) u = soft( u, μ), where soft(, μ) s a soft thresholdng functon wth threshold μ. (c) If mod( km, ) = 0, update the PCA transformaton matrxes P and A n (13) usng the mproved estmate of the k + orgnal HR mage $ ( 1) T ( k+ 1) x = P u. In Algorthm 1, we update P and A n every M teratons to reduce the computatonal complexty. 3. EXPERIMENTAL RESULTS In ths secton, we conduct experments to verfy the effcency of the proposed technque for mage super resoluton. The degraded LR mages are generated by frst applyng a blur kernel and then down-samplng. The blurrng kernel n the smulatons s a 7 7 Gaussan flter wth standard devaton of 1.6. We magnfy the LR mages by a factor of 3, whch s common n the lterature of super-resoluton. We use 5 5 HR mage patches wth overlap of 1 pxel between adjacent patches when transformng HR mages nto adaptve PCA domans. These 5 5 patches are also used to locate the nonlocal smlar neghbors. For color mages, the proposed approach s only appled to the lumnance component and bcubc nterpolator s used for the chromatc components. We compare the proposed approach wth some state-of-the-art mage super-resoluton approaches, ncludng the teratve back-projecton (IBP) [4], the softcuts based method n [8], and the SR based method n [13] 1. The vsual results by these competng approaches are presented through Fg.1 ~ Fg. 3. From these fgures we can see that the IBP method reconstructs the HR mages wth jaggy and chessboard artfacts. The HR mages reconstructed by the SoftCuts based method remove most of such artfacts but they are over-smoothed and many mage detals are elmnated. The approach n [13] s compettve n vsual qualty. However, the reconstructed edges and textures by t are not smooth. Chessboard artfacts and noses can be observed n the edge regons. The reason s that t heavly reles on the tranng data and tends to generate nconsstences between adjacent mage patches. Wthout any exceptons, the proposed approach reconstructs the most vsually pleasant HR mages. The edges and textures reconstructed by our approach are much sharper and cleaner than others. Also, more mage detals are recovered by our approach. 1 We thank the authors of [8] and [13] for provdng ther code or expermental results.

6 In practcal LR magng process, nose s often ntroduced. To demonstrate the robustness of the proposed method to nose, we add the Gaussan noses wth standard devaton of 5 to the smulated LR mages. The HR mages produced by the competng approaches are shown n Fg. 4 and Fg. 5. We can see that the IBP method magnfy the nose snce the back-projecton process s very senstve to nose. The nose s mostly removed by the Softcuts and the SR-based methods n [8] and [13]. However, the mage detals are also removed n ther results. The proposed approach can well handle the super-resoluton and denosng smultaneously. As shown n Fg. 4 and Fg. 5, not only the noses are well suppressed, but also sharp edges and textures are well preserved. The PSNR results of the reconstructed mages n Fg. 1~Fg. 5 are shown n Table 1, from whch we can see that the PSNR values by the proposed method are much hgher that others. 4. CONCLUSION Ths paper presented a sparse representaton (SR) based mage super-resoluton approach by maxmzng the sparsty of the HR mages n adaptve sparse domans. The sparse doman of an mage patch was locally determned by applyng prncple component analyss (PCA) of the nonlocal smlar mage patches. The sparsty of the HR mages was enforced by an l 1 -norm regularzaton term that penalzes the prncple components of the mage sgnals. In addton, a nonlocal smlarty pror of natural mages was also ncorporated to explot the nonlocal mage redundances. The proposed objectve functon wth compound l 1 -norm and l -norm regularzaton terms can be effcently solved by an teratve shrnkage algorthm. Expermental results demonstrated that the proposed approach can reconstruct sharp edges and fne mage detals and s robust to noses. 5. ACKNOWLEDGEMENTS Ths work s supported by the Program for New Century Excellent Talents (No. NCET ), the NSF Chna (No , , and ), and the Ph.D. Program Foundaton of Mnstry of Educaton of Chna (Nos , ). (a) Orgnal (b) Input LR mage (c) Back-projecton [4] (d) SoftCuts [8] (e) Sparse representaton [13] (f) Proposed Fg. 1 Results on the Grl mage wth scalng factor 3.

7 (a) Orgnal (b) Nearest neghbor (c) Back-projecton [4] (d) SoftCuts [8] (e) Sparse representaton [13] (f) Proposed Fg. Results on the Parrot mage wth scalng factor 3. Table 1 The PSNR (db) results of the lumnance components reconstructed by dfferent methods. Images IBP [4] SoftCuts [8] [13] Proposed Grl Parrot Butterfly Nosy Parthenon Nosy Parrot

8 (a) Orgnal (d) SoftCuts [8] (b) Input LR mage (c) Back-projecton [4] (e) Sparse representaton [13] (f) Proposed Fg. 3 Results on the Butterfly mage wth scalng factor 3. (a) Orgnal (d) SoftCuts [8] (b) Input LR mage (e) Sparse representaton [13] (c) Back-projecton [4] (f) Proposed Fg. 4 Results on the nosy Parthenon mage wth scalng factor 3. The standard devaton of Gaussan nose s 5.

9 (a) Orgnal (b) Input LR mage (c) Back-projecton [4] (d) SoftCuts [8] (e) Sparse representaton [13] (f) Proposed Fg. 5 Results on the nosy Parrot mage wth scalng factor 3. The standard devaton of Gaussan nose s 5. REFERENCES 1. X. L and M. T. Orchard, New edge-drected nterpolaton IEEE Trans. Image Process., vol. 10, no. 10, pp , Oct L. Zhang and X. Wu, An edge-guded mage nterpolaton algorthm va drectonal flterng and data fuson, IEEE Trans. Image Process., vol. 15, no. 8, pp. 6-38, Aug X. Zhang and X. Wu, Image nterpolaton by adaptve D autoregressve modelng and soft-decson estmaton, IEEE Trans. Image Process., vol. 17, no. 6, pp , Jun M. Iran and S. Peleg, Moton analyss for mage enhancement: resoluton, occluson and transparency, JVCI., S. Da, M. Han, Y. Wu, and Y. Gong, Blateral back-projecton for sngle mage super resoltuon n Proc. Int. Conf. on Multmeda and Expo, 007, pp , July W. Dong, L. Zhang, G. Sh, and X. Wu, Nonlocal back-projecton for adaptve mage enlargement, n Proc. Int. Conf. Image Process.,009, Oct S. Farsu, M. D. Robnson, M. Elad, and P. Mlanfar, Fast and robust multframe super-resoluton, IEEE Trans. Image Process., vol. 15, no. 1, pp , Jan S. Da, M. Han, W. Xu, Y. Wu, Y. Gong, and A. K. Katsaggelos, SoftCuts: a soft edge smoothness pror for color mage super-resoluton, IEEE Trans. Image Process., vol. 18, no. 5, pp , May J. Sun, J. Sun, Z. Xu, and H. Shum, Image super-resoluton usng gradent profle pror, n Proc. IEEE Computer Vson and Pattern Recognton, pp. 1-8, Jun J. Ca, H. J, C. Lu and Z. Shen, Blnd moton deblurrng from a sngle mage usng sparse approxmaton, n Proc. IEEE Computer Vson and Pattern Recognton, Jun. 009.

10 11. M. Elad and M. Aharon, Image denosng va sparse and redundant representatons over learned dctonares, IEEE Trans. Image Process., vol. 15, no. 1, pp , Dec E. J. Candes and M. B. Wakn, An ntroducton to compressve samplng, IEEE Sgnal Process. Mag., vol. 5, no., pp. 1-30, Mar J. Yang, J. Wrght, Y. Ma, and T. Huang, Image super-resoluton as sparse representaton of raw mage patches, IEEE Computer Vson and Pattern Recognton, vol. 1, pp. 1-8, Jun I. Daubeches, M. Defrse, and C. De. Mol, an teratve thresholdng algorthm for lnear nverse problems wth sparsty constrant, Commu. On Pure and Appled Math., vol. 57, no. 11, pp , Aug P. Combettes, and V. Wajs, Sgnal recovery by proxmal forward-backward splttng, SIAM J. Multscale Model Smul., vol. 4, pp , M. Aharon, M. Elad, and A. Brucksten, K-SVD: an algorthm for desgnng overcompleter dctonares for sparse representaton, IEEE Trans. Sgnal Process., vol. 54, no. 11, pp , Nov L. Zhang, R. Lukac, X. Wu, and D. Zhang, PCA-based spatally adaptve denosng of CFA mages for snglesensor dgtal cameras, IEEE Trans. Image Process., vol. 18, no. 4, pp , Apr L. Zhang, W. Dong, D. Zhang, and G. Sh, Two-stage mage denosng by prncpal component analyss wth local pxel groupng, to appear n Pattern Recognton. 19. A. Buades, B. Coll, and J. M. Morel, A revew of mage denosng algorthms, wth a new one, Multsc. Model. Smulat., vol. 4, no., pp , 005.