A Fast Dictionary Learning Algorithm for Image Denoising Hai-yang LI *, Chao YUAN and Heng-yuan WANG

Transcription

1 08 Internatonal onference on Modelng, Sulaton and Optzaton (MSO 08) ISBN: A ast ctonary Learnng Algorth for Iage enosng Ha-yang LI, hao YUAN and Heng-yuan WANG School of Scence, X'an Polytechnc Unversty, X'an 70048, PR hna orrespondng author Keywords: Sparse presentaton, ctonary learnng, Graph Laplacan, lusterng Abstract he K-SV s one of the well-nown and effectve ethods to learn a unversal and overcoplete dctonary However, K-SV s very expensve because any teraton steps are needed What s ore, when t converts data patches nto vectors for tranng or learnng, K-SV breas down the nherent geoetrc structure of the data o overcoe these ltatons, eployng a subspace partton technque, we propose an effcent and fast algorth, the fast top-botto two-densonal subspace partton algorth, for learnng overcoplete dctonares Experental sulatons deonstrate that our dctonary learnng approach s effectve for age denosng Introducton Sparse and redundant representaton odelng has recently receved extensve research nterest and found successful applcatons n copressve sensng[], age processng tass (copresson, denosng, zoong)[], lnear regresson and varable selecton[3] he sparse representaton proble descrbes that a sgnal can be approxated as a lnear cobnaton of as few as possble bass functons Each bass functon s called an ato and the collecton of the s called dctonary[4] hs dctonary s overcoplete, that s, the nuber of atos s ore than the N denson of each ato More precsely, a target sgnal y R s descrbed as y x, where N M R s an overcoplete dctonary and x s a vector contanng the representaton coeffcent of y We are seeng the sparsest soluton x, the one wth the fewest nonzero entres hs can be forulated as the proble 0 n yx s t x () x where x stands for the so called l 0 0 nor that counts the nuber of nonzero eleents of x, and stands for the axu nuber of nonzero eleents Exact deternaton of sparsest representatons s nown to be an NP-hard proble hus a nuber of algorths have been proposed to provde the sparsest approxaton of a sgnal, ncludng Orthogonal Matchng Pursut (OMP)[5] and Bass N M Pursut (BP)[6] More concretely, gven a tranng data atrx Y R, contanng M sgnals N M N M y R, dctonary learnng s a procedure to fnd a dctonary R he soluton can be obtaned by solvng the followng proble M y x s t x 0 0 X, n Most dctonary learnng algorths perfor two stages[7]: sparse codng and dctonary update In the sparse codng stage, eepng dctonary fxed, X s coputed by solvng () In ctonary update stage, wth a fxed X, the dctonary s updated to reduce the approxaton error he an dfference aong ost dctonary learnng algorths, such as K-SV algorth[8], the ctonary Par Learnng on the Grassann-anfold algorth (PLG)[9] and frst dctonary learnng (L)[0], s the way of updatng the dctonary However, K-SV algorth s expensve In partcular, any teraton steps are needed snce the atos of the dctonary are updated coponent by coponent and the K-SV updates each ato along wth the coeffcents n that ultply t usng 56 ()

2 sngular value decoposton (SV) It s worth notng that the K-SV convert data patches nto vectors for tranng and learnng, and ths converson breas down the nherent structure of the data o overcoe these ltaton, other ethods for dctonary learnng cae up to replace the K-SV or exaple, Lu et al[0] and Zeng et al[] Zeng et al n[] proposed a dctonary par learnng odel (PL odel) for age denosng and desgned a correspondng algorth, called the PLG algorth hs algorth learned an ntal dctonary par by the op-botto wo-densonal Subspace Partton algorth (SP algorth) he ethods of frst dctonary learnng (L) was presented n Lu et al[0] he parttonng procedure n L s equvalent to the frst part of the SP algorth and constructon dctonary s dfferent fro the second part of the SP algorth Motvated by deas n[0] and[], we propose a fast dctonary learnng algorth for age denosng Our ethod s also a two-stage approach that ncludes dctonary learnng stage and sparse codng stage, n whch dctonary learnng stage s dfferent fro the SP algorth n[] and sparse codng stage s slar to the ethod n[] by addng the ntrnsc geoetrc structure of the data through a graph regularzed and usng the learned dctonary to provde a sparse representaton of data patches he paper s organzed as follows In Secton we descrbe the graph Laplacan and then provde a bref descrpton of the graph-based dctonary learnng he correspondng optzaton algorth, a fast top-botto two-densonal subspace partton algorth (SP algorth), s presented n Secton 3 Secton 4 presents soe experent results nally, the paper s concluded n Secton 5 Sparse odng by Learned ctonary and Graph Regularzaton Recall that sparse codng, eepng the fxed learned dctonary, tres to fnd a sparse coeffcent atrx X by solvng () However, ost of the exstng approaches to sparse codng fal to consder the geoetrcal structure of the data space In[], Zheng et al propose a graph based algorth, called graph regularzed sparse codng (GraphS), to learn the sparse representatons that explctly tae nto account the local anfold structure of the data Specfcally, the graph Laplacan s ncorporated nto the sparse codng obectve functon as a regularzer In ths way, the obtaned representatons vary soothly along the geodescs of the data anfold By preservng localty, GraphS can have ore dscrnatng power copared wth tradtonal sparse codng algorths Here we follow deas n Zheng et al[] and ntroduce GraphS n the followng or the gven set of tranng patches { Y }, we construct a weghted undrected coplete graph GV (, EW, ), where the fnte set V of vertces represents the gven patches, V Y,, Y urther, E V V s a set of weghted edges, and these weghts are collected n the weght atrx W R W W by We defne the syetrc weght atrx, Y Y exp for W h h 0 for usng the Gaussan ernel and paraeter h he degree of each vertex Y, the nuber of all edges wth weght to the vertex Y s gven by W, Introducng the dagonal degree atrx dag,,, the graph Laplacan of G s now gven by L W to acheve a sparse atrx L Hence, L s syetrc and postve se-defnte, wth non-dagonal entres beng non-postve, and the su of all entres n each colun (or row) s zero A drect coputaton shows that, r( YLY ) W Y Y W y y Y Y,,,, YY (4) 57

3 easurng the slarty of neghborhood patches n the graph, where we have used the notaton Y Y or each, the vector x s assued to be a good approxaton y Snce the transfor atrx nduces a lnear appng, we can suppose that the vectors x,,, possess a slar topologcal structure as y,,,, and partcularly that, f y and y are -neghbors wth a sall dstance sall herefore, we ncorporate the ter y y, we also have that x x s r( XLX ) W x x = x x,, YY and obtan the new nzaton proble (5) n Y X r( XLX ) X (6) MM NM XR, R where the Laplacan atrx L only depends on the tranng data Y that generates the graph he dctonary learnng algorth by eployng GraphS, called dctonary learnng based on graph regularzaton, s outlned n Algorth Algorth ctonary learnng based on graph regularzaton Input: ranng data Y [ Y ]; Paraeters and Procedures: : opute the graph Laplacan L for the gven tranng set Y : eterne the dctonary by a dctonary learnng algorth based on Y 3: Solve the nzaton proble n Y X r( XLX ) X MM XR 4: Reconstruct the data Y X Loop steps untl the gven nuber of teratons s acheved Output ata Y In the followng, we ntroduce the algorth for solvng the thrd step n Algorth brefly hat s to say, we wll solve the nzaton proble n Y X r( XLX ) X MM XR for gven (nosy) tranng data Y and the dctonary =[ d,, d ], where d col (the vectorzed ) are the dctonary eleents constructed n above We suggest here to solve the proble usng the splt Bregan teraton see eg Goldsten and Osher[3]; Plona and Ma[4] whch s n the consdered case equvalent to the Alternatng recton Method of Multplers(AMM), see Yanelevsy and Elad[5] We outlne the algorth n the followng Algorth Graph regularzed sparse codng Input: ranng data Y [ Y ] ; Laplacan atrx L ; Learned dctonary ; 0 0 X Z B 0 =0 ; Paraeters, u, 0; Nuber of teratons Algorth Iterate untl the gven nuber of teratons s acheved: : opute X as the soluton to ( uix ) XL YuZ ( B ) : opute Z coponentwsely by eployng shrnage z, \ (x, B, ) 3: Update B B Z X Output X 58

4 A ast ctonary Learnng Method Here we wll propose a fast dctonary learnng ethod whch s based on a specal partton tree structure Our ethod s also a two-stage approach that ncludes dctonary learnng stage and sparse codng stage In dctonary learnng stage, we construct the dctonary n two steps We frst obtan a tree structure to partton the set of our tranng patches and then construct the dctonary based on the coputed subset parttons n the tree, usng a fast top-botto two-densonal subspace partton algorth (SP algorth) he frst step, the tree constructon, s dfferent fro the frst part of the SP algorth, whle the second step, constructng the dctonary, s equvalent to the second part of ethod SL In sparse codng stage, we add the ntrnsc geoetrc structure of the data through a graph regularzed and use the learned dctonary to provde a sparse representaton of data patches he SP algorth s outlned n Algorth 3 Step: onstructon of the partton tree or the gven tranng set,, n n Y Y R of age patches We copute the ean nn : Y R (7) and the two non-syetrc (n n) covarance atrces : ( Y )( Y ) L : ( Y ) ( Y ) (8) R Now, the noralzed egenvectors u, u andv, and R s coputed By the frst two egenvalues we refer to the two largest egenvalues v correspondng to the frst two egenvalues of L u: argaxx Lx, x v: argaxx Rx, (9) x representng the an structures of the tranng patches beng not captured by the ean patch We copute the nubers s uyv, s uyv,,, (0) hese nubers { s,, } and { s,, } gve us a easure, how strong each sngle patch s correlated to the found structure and wll be used to obtan a partton of the set of all patches { Y } nto four partal sets rst, n the frst level, we partton the one-densonal real nuber { s,, } nto two clusters { s } and { s } by K-eans, n whch {,, } and hen, n the second level, we partton the one-densonal real nuber sets { s } and { s } nto two clusters { s }, { s } and { s }, { s } by K-eans respectvely, n whch, and, herefore, { Y }= { Y} { Y} and { Y}={ Y} { Y}, { Y}={ Y} { Y} In ths way, we can dvde two level tree structure wth every calculaton Rears Havng found ths frst partton, we can proceed to partton the obtaned subsets further, usng the sae schee hs procedure yelds a bnary tree, where we stop the further partton of a subset, f t contans two tranng data that autoatcally separate the two classes Snce two level tree structures are obtaned n every calculaton, the algorth of our artcle speeds up SP algorth 3 If we yeld a bnary tree wth the frst three egenvalues or ore egenvalues n above procedure, then the algorth has a better convergence rate but a weaer perforances such as structural slarty ndex (SSIM), pea sgnal to nose rato (PSNR) and root ean square error (RMSE) n general 59

5 Step : eterne the dctonary fro the partton tree Each not n the tree s now assocated wth a subset of tranng patches { Y }, where {,, } denotes the subset of ndces of these patches We assue that the root of the tree has the not nuber (e = {,, } ) and we proceed nuberng by gong through each level fro left to rght or each not, we copute the ean (center) Y () and the noralzed sngular vectors to the axal sngular value of and, e u : argax x x v : argax x (3) x If denotes the axal sngular value of then uv s the best ran- approxaton of, snceu and v are the frst vectors n the sngular value decoposton of he dctonary s now deterned as follows We fx the frst dctonary eleent uv (4) capturng the an structure of the ean urther, for each par of chldren nots and to the sae parent wth eans and +, we set u v u v (5) :, : thereby capturng the dfference of an structures of,,, ), we construct the dctonary =[ d,, d ] and + Let d col (the vectorzed Algorth 3 (SP algorth) ast top-botto two-densonal subspace partton algorth Input: ranng age patches, the axu depth of the bnary tree Procedures: : he frst node s the root node ncludng all age patches : or all age patches n the current leaf node, run the followng )-4) steps: ) opute egenvectors u, v and u, v correspondng to the frst two egenvalues of the two covarance atrxes ) opute the two-densonal proecton representatons of all age patches fro ths node, that s, s u Yv and s u Yv,,, 3) Partton the one-densonal real nuber set s nto two clusters { s } and { s } by K-eans hen partton the age patches correspondng to these two clusters nto the left chld { Y } and the rght chld { Y } Sultaneously the depth of the node s added one 4) Partton the one-densonal real nuber sets { s } nto two clusters { s } and { s } by K-eans hen partton the age patches { Y } correspondng to these two clusters nto the left chld { Y} and the rght chld{ Y } Partton the age patches { Y } n an analogous anner Sultaneously the depth of the node s added one 3: I the depth of the node s larger than the axu depth or the nuber of age patches n ths leaf node s saller than the row nuber or colun nuber of the age patches, HEN stop the partton ELSE repeat Step recursvely for the left chld node and the rght chld node 60

6 4: opute the dctonary for each node by the followng )-3) steps: ) opute the center and the noralzed sngular vectors u and v to the axal sngular value of and n the root node onstruct the frst ato of dctonary uv ) or each par of chldren nots and to the sae parent wth eans and +, copute the noralzed sngular vectors u, and + sngular value 3) opute the ato :, where v and u +, v +to the axal : uv u v 5: ollect the ato of all leaf nodes nto the dctonary d d vectorzed ) Output the dctonary,,, where d col (the Experents In ths secton, we present experents to evaluate the dctonary perforance of our proposed algorth copared wth other algorths ntroduced n the paper In the part, we present experental result, wth the a of tranng a dctonary for sparsely representng natural age patches We then turn to test the age denosng perforance of the dctonary learned by our approach Our sulatons were perfored n MALAB R00b envronent on a syste wth 38 GHz PU and 4 GB RAM, under Mcrosoft Wndows 7 operatng syste As a rough easure of coplexty, we wll enton the runnng tes of the algorths We show the proveent acheved by applyng the above ethods to the age denosng proble In ths set of experents, we eploy the ethodology[8] gven by Elad We choose three well-nown ages of sze as test ages, ncludng Barbara, Boat, House, Each age s contanated by artfcally addng zeros-ean whte Gaussan nose at fve dfferent varances An obectve age qualty etrc plays an portant role n age denosng applcatons urrently, three classcal age qualty assessent etrcs are typcally used: the Pea Sgnal-to-Nose Rato (PSNR) and the easure of Structural Slarty (SSIM) We cut the nosed age nto sall patches of sze 8 8 he regularzaton paraeters have been eprcally chosen to be 6; = 30 ; u 005 It s also noted that the denosed age s obtaned wth an average constant 6

7 able PSNR values of the denosed results Iage Barbara =5 =0 =5 =0 =5 PSNR IME PSNR IME PSNR IME PSNR IME PSNR IME Algorth K-SV L Ours Iage Boat =5 =0 =5 =0 =5 PSNR IME PSNR IME PSNR IME PSNR IME PSNR IME Algorth K-SV L Ours Iage House =5 =0 =5 =0 =5 PSNR IME PSNR IME PSNR IME PSNR IME PSNR IME Algorth K-SV L Ours able SSIM values of the denosed result Iage K-SV PS Ours Barbara Boat House

8 gure splayng denosng results for, (a) orgnal age (b) Nosy age, (c) Iage denosed usng K-SVdctonary (d) Iage denosed usng L dctonary(e) Iage denosed usng Our dctonary able presents the fnal denosng PSNR results obtaned fro K-SV, L algorths wth addtonally the fxed ours ethods he SSIM results of the three test ethods are reported n able gure dsplays the orgnal, nosy and denosed Barbara, Boat, House, ages for nose level =0 Based on these results, we can observe that our proposed algorth, SP algorth, and L algorth n general not only cost less te but also provde hgher PSNR result and SSIM values n age denosng copared wth the K-SV algorth Although SP algorth and L algorth have the slar results n general, t s notceable that SP algorth needs less te copared wth L Suary In the paper, we present a fast top-botto two-densonal subspace partton algorth (SP algorth) for learnng overcoplete dctonary, whch s based on a specal partton tree structure In constructon of the partton tree step, our algorth can obtan two level tree structures n every calculaton, and hence t costs less te than the frst part of the SP algorth and L It s equvalent to the second part of ethod SL n constructon dctonary step Experental results on synthetc data and age patches llustrate that SP and L not only have hgher qualty but also cost less te than K-SV, and that at the sae te, SP needs less te than L In the future, we wll consder provng SP algorth and applyng t to other applcatons Acnowledgeent he authors would le to than edtoral and referees for ther coents whch help us to enrch the content and prove the presentaton of the results n ths paper he wor was supported by the Natonal Natural Scence oundatons of hna (797) and the Scence oundatons of Shaanx Provnce of hna (05JM0) 63

9 References [] L onoho, opressed sensng, IEEE rans on Inforaton heory 5(4) (006) [] M Elad, MA gueredo and Y Ma, On the role of sparse and redundant representatons n age processng, Proceedngs of the IEEE, 98 (6) (00), [3] R bshran, Regresson shrnage and selecton va the lasso, J Royal Statst Soc B 58() (006) [4] S Mallat and Z Zhang, Matchng pursuts wth te-frequency dctonares, IEEE rans on Sgnal Proc 4() (993), [5] Y Pat, R Rezafar, P Krshnaprasad, Orthogonal atchng pursut: Recursve functon approxaton wth applcatons to wavelet decoposton, n Proceedngs of the 7th Asloar onference on Sgnals, Systes & oputers (993) [6] S hen, onoho, M Saunders, Atoc decoposton by bass pursut, SIAM Revew 43() (00), 9-59 [7] W a, Xu, W Wang, Sultaneous codeword optzaton (SO) for dctonary update and learnng, IEEE rans on Sgnal Process 60() (0) [8] M Elad, M Aharon, Iage denosng va sparse and redundant representatons over learned dctonares, IEEE rans on Iage Process 5 (006) [9] X Zeng, W Ban, W Lu, J Shen and ao, ctonary par learnng on Grassan anfolds for age denosng, IEEE rans on Iage Processng 4 () (05) [0] L Lu, J Ma, G Plona, Graph regularzed sesc dctonary learnng, [] X Zeng, W Ban, W Lu, J Shen and ao, ctonary par learnng on Grassan anfolds for age denosng, IEEE rans on Iage Processng 4 (05) [] M Zheng, JJ Bu, hen and Wang, Graph regularzed sparse codng for age representaton, IEEE rans on Iage Processng 0(5) (0), [3] Goldsten and S Osher, he splt Bregan ethod for L-regularzed probles, SIAM J Iagng Scences () (009) [4] G Plona and J Ma, urvelet-wavelet regularzed splt Bregan teraton for copressed sensng, Int J Wavelets Multresolut Inf Process 9() (0) 79-0 [5] Y Yanelevsy and M Elad, ual graph regularzed dctonary learnng, IEEE ransactons on Sgnal and Inforaton Processng over Networs (4) (06)