A Fast Image Restoration Method Based on an Improved Criminisi Algorithm

Transcription

1 A Fast Image Restoration Method Based on an Imroved Algorithm Yue Chi1, Ning He2*, Qi Zhang1 Beijing Key Laboratory of Information Services Engineering, Beijing Union University, Beijing , China. 2 College of Information Technology, Beijing Union University, Beijing , China. 1 * Corresonding author. Tel.: ; xxthening@buu.edu.cn Manuscrit submitted Aril 18, 2016; acceted July 29, doi: /jc Abstract: rooses an imroved image restoration algorithm that roduces better reairs and reduces the comutational time. First, we imroved the riority calculation and included a ste that transforms the original confidence term into an index to achieve a more recise reair. Second, in large damaged areas of an image, we use a local searching method to find the otimal matching block to seed u the reair rocess. Our exerimental results show that the imroved method significantly increased the seed of the method, effectively retained image structures, and roduced better visual effects. Key words: Texture synthesis, image restoration, riority, local matching. 1. Introduction Image restoration refers to reconstructing damaged images or removing excess. Bertalmio [1] first roosed that image restoration can be reduced to a mathematical exression, which can then be automatically solved using a comuter. Image restoration has become a major research area in comuter grahics and comuter vision. It has significant alications such as reairing cultural relics, film and television ost-roduction secial effects, virtual reality, and removing unwanted objects. There are currently two tyes of classic image reair techniques. The first uses structural rehabilitation methods such as the BSCB model (M. Bertalmio-G. Sairo-V. Caselles-C. Ballester, BSCB) [2], model (Total Variation, ) [3], and the CDD model (Curvature Driven Diffusion, CDD) [4]. models reair textural synthesis, and include the model. Structural rehabilitation methods require large numbers of calculations, are time consuming, and result in more obvious blurring when considering large areas. Thus, they are more suitable for roblems such as small scratches and stains [5]-[8]. Textural synthesis reair methods use ixel block unit feature extraction methods within a known area and select the best matching block. The synthesis is alied to the damaged area and is more suitable for large areas [9]. In 2003, et al. [10] roosed an image restoration method based on texture. The largest restoration riority is given to a ixel block with a defect in the border of the source area. This method searches for a block that is most similar to the current ixel block, and then uses it to relace the current ixel block and comlete the reair of the damaged area. This algorithm is based on block texture units but not ixel units, so the reair is more time consuming. The algorithm can satisfactorily reair large damaged areas, but the riority calculation and deficiencies in the matching block selection rocess can have a negative imact on the reair. 591

2 Many researchers have attemted to solve these disadvantages of the algorithm. Bing et al. [11] increased the border riorities to imrove the method, and selected different arameters for different tyes of images. These changes did imrove the reair to some extent, but increased blurring. Wang et al. [12] roosed a robust exemlar-based image reair method. Their model used a regularisation factor to adjust the reair block riority function. They considered the SSD (sum of squared differences) of the modified values and the NCC (normalised cross correlation) to search for the best matching block. This method roduced better reairs, but involves comlicated calculations and is time consuming. Wong et al. [13] roosed determining the matching blocks based on the varied degree of similarity between the block and the block to be reaired. This algorithm can achieve an excellent reair, but it is time consuming and is too deendent on the similarity of the reair block and samle blocks. In the same year, Lei et al. [14] roosed a fast image restoration algorithm based on an adative temlate and the algorithm. It first considers the reair oint gradients with resect to neighbouring ixels to effectively estimate the oint to be reaired. When considering the direction of the illumination line, the illumination characteristics along the line to be reaired adatively determine the size of the temlate. The algorithm erforms well with resect to edge details and smooth regions. For large areas, the reair will always be a discontinuity. Zhou et al. [15] analysed the information structure that must be reaired with resect to the intensity of the image block. Their imroved algorithm can erfectly reair structural edges. These studies were all based on the algorithm and focused on reairing textural information. In this aer, we roose an imroved riority calculation algorithm, which recisely restores the confidence and image data using a joint decision image sequence. This method is quicker than existing techniques, because it uses a local search method to find the otimal matching block. Our exerimental results show that the imroved method significantly reduced the comutational time and effectively reaired the structure of the image. 2. Algorithm The algorithm for texture synthesis is a classical algorithm for reairing large areas. It combines the advantages of texture synthesis and image reair techniques. The basic rincile of the algorithm is shown in Fig. 1. I is an image containing a damaged area that we wish to reair, is the damaged area, is the border region, is a known source area of the image, P is a unit ixel boundary, is considered the centre of P, the target block. I is the tangential direction of the illumination line and n is the damaged boundary tangent vector method. Fig. 1. Basic schematic of the algorithm. The algorithm is described as follows. We first calculate the riority of the block. The riority deends on the combination of two values, the confidence C ( ) and the data D( ). C ( ) corresonds to the centre ixel ( ) of the atch block ( ), which is roortional to the original image. A larger C ( ) corresonds to a greater riority, which reflects that a region containing more of the original 592

3 information should be given riority. D( ) reresents the normal n, which belongs to the boundary ( ) of oint and the intact area of the roduct of the edge gradient vector, I. A larger value of D( ) corresonds to a higher riority, which reflects that the boundary of the original atch has a significant relationshi with the structural strength, and should be given riority. The riority formula of the algorithm is P( ) C ( ) D( ), where C ( ) is the confidence term defined as C ( ) q ( I ) C (q) (1) and D( ) is the data term defined as D( ) I n. (2) Here, is the area of the temlate of, that is, the total number of ixel temlate oints. C (q) is the confidence value for ixel oint q, which must satisfy the initial conditions 1 q C (q) 0 q I is the normalised factor;. (3) =255 3 for a 24-bit RGB image. Second, we search for the best matching block and use it for the reair. The algorithm is a global search method. It searches in undamaged areas for similar blocks to use to reair the damage. To reair samle, we use q and the sum of squares difference d c (, q ) to find the colour of the samle with the minimum sum of squares difference, which we use as the best matching module,. The reair ixels in are filled with the corresonding q ixels in. This simultaneously reairs the textural information and structural characteristics. The q minimum sum of squares difference (SSD) is arg min d c (, q ), q q (4) and we calculate the colour sum of square differences using dc (, q ) [( I R I R ) 2 ( I G I G )2 ( I B I B )2 ]. (5) Here, I and I corresond to ixel oints in and q, resectively. Third, we udate the confidence value. Each reair block is at the edge of the area to be reaired, and the edges of the reair area constantly change after a texture block is reaired. When the highest riority target block is reaired, its confidence is udated using C ( ) C ( ),. We reeat these three stes until the reair is comlete. The algorithm erforms well but it is time consuming, and the riority calculation and matching 593

4 block selection are not otimal. The algorithm is based on the texture units of blocks rather than ixel units, so it takes longer to reair large damaged areas. The riority calculation used to select areas and method of finding matching blocks affect the texture of the damaged areas, which affects the accuracy. Imroved riority formulas can reduce the reair time. We roose using a local range search for the best matching block to significantly imrove the comutational seed. 3. Imroved Algorithm 3.1. Priority Calculation The key idea of the roosed algorithm is to consider restorative effects of the structure and the texture in terms of the order of the reair blocks. The riority order must deend on two factors: the temlate window, i.e. the confidence C ( ), and structural features around the area to be reaired, i.e., the data items D( ). The algorithm s riority formula is P( ) C ( ) D( ). During the image restoration rocess, the confidence gradually reduces to zero, leading to incorrect filling orders. We roose using the riority P( ) C ( ) D( ), where 1. According to (1), a larger undamaged roortion of the block to be reaired corresonds to a larger confidence. According to (2), if the filling oint is nearer the edge, I is larger and the angle between I and n is smaller. Then, D( ) is larger and the area will be reaired sooner. The light intensity of ortions of the image s linear structure determines the size of data items. Edges of the image are reaired first to retain the structure and reduce diffusion. Therefore, D( ) has more of an influence on the riority. The image edges and gradients reflect the image texture features. Pixels with large gradients reresent a rich textured image. To increase the gradient transform (that is, increase the influence of D( ) by weakening C ( ) ), we set C ( ) q ( I ) C (q) 1. Then, C ( ) C ( ), the confidence decreases, and the imortance of the data items increases. This imroves the textural details, and increases the accuracy of the reair. In our exerimental results, when gradually increased from 2 the reair was more effective, the otimal results where for 3, and the result was inaccurate when = Local Area Search for the Otimal Matching Block The original algorithm searches the entire image to find the most similar block. But many similar blocks are in the vicinity of the target area, so a global search reduces the efficiency of the algorithm. To reduce the search sace and achieve satisfactory reairs, we use a local area search. Fig. 2. Local search for matching block. Fig. 2 shows an image containing a damaged area. is the damaged area, is the border region, 594

5 is a known source area, P is a unit ixel boundary, and is the centre of P (the target block). I is the tangential direction of the illumination line, and n is the damaged boundary tangent vector. In the algorithm, we set the block size of the atch (red box) to a 9 9 ixel block, and search through all the matching blocks in the entire undamaged area. This is time consuming, and reduces the reair efficiency. However, the most similar blocks are tyically located in the general vicinity of the block to be reaired. We determine the ixel with the highest riority, and then extend down 13 ixels (i.e., in a local ixel block). With this in mind, the roosed method uses a local search block matching method to solve the original algorithm and reduce the comutational time. Fig. 2 shows the basic rinciles of the local search method. It uses the following stes. 1) The current ixel has the highest riority. Extend down 13 ixels, within the scoe of a artial ixel block to find the best matching block. 2) Search within the ixel block using a 9 9 samle temlate. That is, start in the uer left corner from the first oint, and using a 9 9 ixel temlate traverse from left to right, down one line, and continue to traverse from left to right, until you have traversed all the ixels. 3) After reairing this section of the image using the best matching block, udate the boundary information and return to Ste (2) until the entire image is reaired. 4. Results and Analysis We ran our exeriment using a 3.4GHz rocessor, 8GB of memory, and MATLAB R2010b. We alied the algorithm to natural and character image restoration, and single and multitarget removal, to illustrate its adatability and robustness. The image dimensions were , , before alying the algorithm and the images were jg files. After alying the algorithm, the image dimensions were,, and there were two grous. We evaluated our algorithm in terms of the comutational time and the eak signal to noise ratio (PSNR), that is, PSNR 10 lg( 2552 ), MSE where MSE reresents the difference between the original and restored image [16]-[19]. If I i are the original ixel values, J i are the reaired the ixel values, and N is the number of ixels, N MSE (I i 1 i J i )2. N Although PSNR is not a standard evaluation for small-scale defect image restoration, it can reflect the effect of the reair [20]-[23] The Imroved Algorithm Results before and after Comarison and Analysis Fig. 3 illustrates the results when reairing a manual obstruction. The image has a simle structure and minimal colours, so the results of the two algorithms are similar. Fig. 4 shows the removal of some targets from an image. The image had many targets that were distributed across regions of different colours, so this image was more difficult to reair [24]. The algorithm s result was not ideal; there were green ixels throughout the blue lake area. The results of the roosed algorithm were more successful. Fig. 5 contains a scratch. The results obtained by algorithm showed an affective background reair, but there were white lines on the image due to the scratches. The roosed algorithm roduced better results. 595

6 Table 1 comares the run times of the algorithms. The comutational time of the roosed algorithm deended on the size of the image, but was significantly less that the original algorithm. The roosed algorithm erformed better in terms of the PSNR than the algorithm (Table 2), and roduced better visual imrovements. (a) (b) (c) Fig. 3. Image reair for a subjective obstruction. (a) (b) (c) Fig. 4. Multitarget removal. (a) (b) (c) Fig. 5. Photo restoration to remove crease. (d) (d) (d) Table 1. Comarison of Running Times Fig. 2 Fig. 3 Fig /s /s Table 2. Comarison of Peak Signal to Noise Ratio Fig. 2 Fig. 3 Fig Exerimental Results Comared to Control and Analysis To demonstrate the effectiveness of the imroved algorithm, we comared it with the algorithm, algorithm, and algorithm [25], [26]. We considered edge restoration, regular texture restoration, checkerboard mask corrosion restoration, and stried mask corrosion restoration. We used the rogram running time and eak signal to noise ratio (PSNR) to measure the efficiency of the algorithm. In our control exeriments, we set the model algorithm [27], [28] to use 500 iterations Analysis of edge and regular texture regions The edge and regular texture areas restoration exeriments verified that the imroved algorithm can 596

7 effectively maintain object connectivity and comliance in terms of a subjective evaluation. The edges of Fig. 6 Fig. 9 show that the roosed method and the algorithm outerformed the other algorithms. The PSNRs for the roosed method were higher and a visual evaluation shows that the reair was the most effective. The algorithm result is artially smoothed and has oor connectivity (Fig.6), and breaks some edges (Fig. 7) roducing obvious blurring. The algorithm was time consuming and did not roduce otimal results. In Fig. 6, we can see that the distorted the image. For the regular texture reair examles, the roosed method roduced better results and more comletely restored the image, while the other three methods have varying degrees of defect. Fig. 6. Reairing edges (boat). Fig. 7. Reairing edges (hat). Table 3. Comarison of Running Times Fig. 6 Fig. 7 Fig. 6 Fig. 7 size /s /s /s /s Table 4. Comarison of Peak Signal to Noise Ratio (PSNR) size Fig. 8. Reairing regular texture regions (table cloth). Fig. 9. Reairing regular texture regions (brick). Table 5. Comarison of Running Timed size Fig Fig

8 Table 6. Comarison of Peak Signal to Noise Ratio Fig. 8 Fig. 9 size Checkerboard and Strie Mask Image Restoration Our checkerboard and stried mask corrosion restoration exeriments verified that the imroved algorithm can effectively restore the comlex structure of an image in a subjective evaluation. The results for the checkerboard mask are shown in Fig. 10 and Fig. 11, and the results for the stried mask are shown in Fig. 12 and Fig. 13. The roosed algorithm roduced the best results in terms of the PSNR and a visual evaluation. The checkerboard results in Fig. 10 and Fig. 11 are relatively similar, but the roosed algorithm erforms subjectively better. The stried mask results in Fig. 12 and Fig. 13 show a large loss in image information, so the reairs were challenging. Our results show that the other three methods roduce blurring. The eyes and mouths in the Lena and Barbara images are not clear. However, the roosed method effectively restored the image information, and the results had a high signal to noise ratio. Fig. 10. Reairing an image with a checkerboard mask (Lena). Fig. 11. Reairing an image with a checkerboard mask (Barbara). Table 7. Comarison of Running Time Fig. 10 Fig. 11 /s /s /s /s Table 8. Comarison of Peak Signal to Noise Ratio Fig. 10 Fig Fig. 12. Reairing an image with a stried mask (Lena). 598

9 Fig. 13. Reairing an image with a stried mask (Barbara). Table 9. Comarison of Running Time Fig. 12 Fig. 13 /s /s /s /s Table 10. Comarison of Peak Signal to Noise Ratio Fig. 12 Fig Conclusion roosed an imrovement to the original image restoration algorithm that uses a different riority calculation and search rocess. The roosed method imroves the texture information and retains more structural information to avoid diffusion. When searching for the otimal matching block, we used local search methods to imrove the accuracy and reduce the comutation time, thereby increasing the efficiency of image reair rocess. Our exeriments show that the algorithm can effectively reair comlex images and has a broad range of alications such as reairing old hotos and multitarget removal. Acknowledgment This work was suorted by the National Natural Science Foundation of China (Grant Nos , , , U ), Beijing Municial Natural Science Foundation (Grant Nos , ). References [1] Bertalmio, M., Sairo, G., Caselles, V., et al. (2000). Image inainting. Proceedings of ACM SIGGRAPH ( ). [2] Bertalmio, M., Caselles, V., Rougé, B., et al. (2003). Based image restoration with local constraints. Journal of Scientific Comuting, 19(1-3), [3] Chan, T. F., & Shen, J. (2002). Mathematical models for local nontexture inaintings. Siam Journal on Alied Mathematics, 62(3), [4] Chan, T. F., & Shen, J. (2001). Non-texture inainting by curvature-driven diffusions (CDD). J. Visual Comm. Image Re., [5] Efros, A. A., & Leung, T. K. (1999). Texture syntesis by non-arametric samling. IEEE International Conference on Comuter Vision, [6] Drori, I., Cohen-Or, D., & Yeshurun, H. (2003). Fragment-based image comletion. Acm Transactions on Grahics, 22(3), [7] Zhang, H. Y., & Peng, Q. C. (2007). A survey on digital image inainting. Journal of Image & Grahics, 12(1), [8] Liang, L., Liu, C., Xu, Y. Q., et al. (2001). Real-time texture synthesis by atch-based samling. Acm 599

10 Transactions on Grahics, 20(3), [9] Dai, S. M., Zhang, H. Y., & Zeng, C. (2010). A fast algorithm of exemlar based image comletion. Microcomuter & Its Alications, (22), [10], A., Perez, P., & Toyama, K. (2003). Object removal by exemlar-based inainting. Proceedings of IEEE Comuter Society Conference on Comuter Vision & Pattern Recognition ( ). [11] Bing, H. S., Lin, Z. X., Yun, X. Y., et al. (2011). An imroved image inainting algorithm based on texture synthesis. Journal of Hefei University of Technology, 34(2), [12] Wang, J., Lu, K., Pan, D., et al. (2014). Robust object removal with an exemlar-based image inainting aroach. Neurocomuting, 123, [13] Wong, A., & Orchard, J. (2008). A nonlocal-means aroach to exemlar-based inainting. Proceedings of ICIP International Conference on Image Processing, [14] Lei, Q. U., Wei, S., Liang, D., et al. (2008). A fast image inainting algorithm by adative mask. Journal of Image & Grahics, 13(1), [15] Zhou, Y., Li, L., & Xia, K. (2012). Research on weighted riority of exemlar-based image inainting. Journal of Electronics, 29(Z1), [16] Liang, Z., Yang, G., Ding, X., et al. (2015). An efficient forgery detection algorithm for object removal by exemlar-based image inainting. Journal of Visual Communication & Image Reresentation, 30(C), [17] Kim, B. S., Kim, J. S., & Park, J. (2015). Exemlar based inainting in a multi-scaled sace. Otik-International Journal for Light and Electron Otics, 126(23), [18] Ren, D., Zhang, H., Zhang, D., et al. (2015). Fast total-variation based image restoration based on derivative alternated direction otimization methods. Neurocomuting, 170, [19] Suryanarayana, M. Muddala, M. S., & Roger, O. (2016). Virtual view synthesis using layered deth image generation and deth-based inainting for filling disocclusions and translucent disocclusions. Journal of Visual Communication and Image Reresentation, 38, [20] Jiao, A. S. M., Tsang, P. W. M., & Poon, T. C. (2015). Restoration of digital off-axis Fresnel hologram by exemlar and search based image inainting with enhanced comuting seed. Comuter Physics Communications, 193, [21] Li, Y., Jeong, D., Choi, J. I., et al. (2015). Fast local image inainting based on the Allen Cahn model. Digital Signal Processing, 37(1), [22] Zhong, M., & Qin, H. (2015). Surface inainting with sarsity constraints. Comuter Aided Geometric Design, 41, [23] He, N., Wang, J. B., Zhang, L. L., et al. (2016). Convex otimization based low-rank matrix decomosition for image restoration. Neurocomuting, 172(C), [24] So, J., & Kim, B. (2015). A fast exemlar-based image inainting method using bounding based on mean and standard deviation of atch ixels. Ieice Transactions on Information & Systems, [25] Elad, M., Starck, J. L., Querre, P., et al. (2005). Simultaneous cartoon and texture image inainting using morhological comonent analysis (). Alied & Comutational Harmonic Analysis, 19(3), [26] Zhang, T., & Hong, W. (2010). Image inainting based on featured adative dictionary selection. Otical Technique, 36(5), [27] Ye, X., & Zhou, H. (2013). Fast total variation wavelet inainting via aroximated rimal-dual hybrid gradient algorithm. Inverse Problems & Imaging, 3(3), [28] Afonso, M. V., & Sanches, J. M. R. (2015). A total variation recursive sace-variant filter for image denoising. Digital Signal Processing, 40(C),

11 Yue Chi was born in Beijing, China in She received her B.S. degree in electronic information engineering in 2013 at Beijing Union University, Beijing, China. Currently, she is a ostgraduate student from Beijing Key Laboratory of Information Services Engineering, Beijing Union University. Her research interests include digital image inainting and digital image restoration. Ning He was born in Panjin, Liaoning Province in She graduated from the Deartment of Mathematics at Ningxia University in July 1993.She received M.S. and Ph.D. degree in alied mathematics from the Northwest University and Caital Normal University in July 2003 and July 2009, resectively. Currently she is a rofessor of the Beijing Union University. Her research interests include image rocessing and comuter grahics. Qi Zhang was born in Xingtai, Hebei in She received her B.S. degree in communication engineering in 2014 at Tangshan College, Hebei, China. Currently, she is a ostgraduate student from Beijing Key Laboratory of Information Services Engineering, Beijing Union University. Her research interests include digital image recognition and digital image segmentation. 601