JOINT IMAGE REGISTRATION AND EXAMPLE-BASED SUPER-RESOLUTION ALGORITHM

Transcription

1 JOINT IMAGE REGISTRATION AND AMPLE-BASED SUPER-RESOLUTION ALGORITHM Hyo-Song Kim, Jeyong Shin, and Rae-Hong Park Department of Eectronic Engineering, Schoo of Engineering, Sogang University 35 Baekbeom-ro, Mapo-gu, Seou, 11-74, Korea ABSTRACT Supper-resoution (SR) methods are cassified into two different methods: image registration ()-based methods and exampe-based methods. The proposed oint SR method is focused on estimating highresoution (HR) video sequences from ow-resoution (LR) ones by combining the two different methods. The -based SR method coects information from adacent frames to reconstruct HR images in the video sequence. Exampe-based SR methods give good textures and strong edges in the resut HR video. In this paper, -based and exampe-based SR methods are fused based on the gradient features. The proposed oint SR method gives smaer peak signa to noise ratio than the exampe-based method, however it shows better reconstruction resuts on high-eve features such as characters in images. Experimenta resut of the proposed oint SR method shows ess noise and higher contrast than the exampe-based method. KEYWORDS Super-Resoution, Image Registration, Motion Estimation, Motion Compensation, Exampe-Based Learning, Sparse Coding, Neigborhood Regression. 1. INTRODUCTION Supper-resoution (SR) methods are traditionay cassified into two different casses: image registration ()-based and exampe-based methods [1]. is the task of finding the motion between two or more frames of the same scene. It is noted that the motion estimated by does not necessariy describe the true motion of either the camera or the obect in a scene. The most common way to aow a reiabe impementation of the is to estimate the motion using ony two-dimensiona transations, under the assumption of sma motion within the scene. In [1] more eaborated methods are presented incuding methods using rotation and scaing, basicay intended for scanned documents. -based SR methods can be cassified into two main approaches: featured-based methods and bock-based methods. The first approach fits a motion mode by using a sparse set of point correspondences, whie the second one uses the information of the entire pixes within search area. Feature-based SR methods can cause fase matches due to their regression nature and require a arge number of points in order to achieve a high eve of accuracy. On the contrary, the bockbased methods can obtain the motion information with a good accuracy for producing a highresoution (HR) image. Whie the feature-based methods use a sma number of points, the bockbased methods utiize a the overapping bocks of the adacent images. A ot of bock-based methods estimate image motion by minimizing a cost function between two motion-corrected 47

2 images [], however suffer from a very high computationa cost. Athough SR methods which use motion information between adacent frames have been successfu on some of the video sequences, they are not suitabe for the video sequences with stationary obects and background, which do not contain information to fi in the sub-pixes. Moreover, estimating a true motion vector in spatiay high-frequency regions and handing occuded regions is not a simpe task. Thus, SR techniques on a singe image are sti important even though the goa is to perform SR on video sequences. Most singe image SR methods use exampe-based approach, which earns from exampe images to form a dictionary and appy it to each image patch. Xiong et a. [3] used the soft information and decision on the one-to-many correspondence of dictionaries to sove the dimensionaity gap between ow-resoution (LR) and HR spaces. Timofte et a. improved the conventiona sparse coding method by introducing anchored neighborhood regression (ANR) method [4] [6]. Zhu et a. [7] used optica fow on image patches to form deformabe patches, in order to make the earned dictionary more expressive.. RELATED WORK.1. -Based SR Methods -based SR methods [1] [] generay concentrate on spatia domain and typicay consist of two processes: and HR reconstruction. is the process that cacuates the motion parameters based on a specific motion mode. HR reconstruction incorporates the estimated motion parameters into inverse estimation. The aiasing effect among LR images may reduce the accuracy of registration. Accurate registration is a chaenging task because motion parameters are cacuated from a number of aiased LR images. To dea with the probem, various SR agorithms have been proposed to reduce the registration error on the fina estimated HR image... Dictionary-Based SR Methods A arge number of dictionary-based SR methods [3] [8] have been deveoped for the ast decade. They use dictionary of image patches or patch-based atoms that are trained to represent natura image patches efficienty, which brings advantages in both time compexity and accuracy of the SR resuts...1. Neighbor Embedding Methods Neighbor embedding methods are generay used in patch-based methods. They assume that the LR input patches can be approximated by a weighted sum of HR trained patches. LR and HR patches are earned simutaneousy in the training phase. A typica method, ocay inear embedding [3], can be written as K x x ', (1) where x denotes the resut HR patch and x ' i represents the i th candidate HR patch, which is the pair of the i th nearest neighbor LR patch of the input LR patch. K is the number of neighbors in consideration and w i denotes the weight of the i th candidate HR patch. i 1 w i i 48

3 ... Sparse Coding Methods Unike neighbor embedding methods, sparse coding methods try to find efficient representations, dictionary atoms, of the image patches. Zeyde et a. [8] buit an efficient and improved method to train a sparse dictionary. They buit dictionaries for both LR patches and their corresponding HR patches by using oint optimization on these two patches. After LR and HR dictionaries are trained, a sparse representation α of an input LR image patch can be cacuated as α arg min α D α y α, () 1 where D denotes the LR dictionary, α is a sparse representation of LR patch, y represents the input LR patch, and denotes the reguarization parameter which contros the significance of the sparsity constraint ( 1-norm in the second term) over the modeing error ( -norm in the first term)...3 ANR Method ANR method [4] reformuates the dictionary of Zeyde et a. [8] to pre-compute regression matrices used to cacuate the resut HR patches. Instead of using LR and HR dictionaries directy, it considers the nearest neighbors of a specific th dictionary atom d, which is the th coumn vector of the LR dictionary D. Using ony the nearest neighbors of d, a oca neighborhood LR dictionary N is cacuated aong with its corresponding oca neighborhood HR dictionary N. Instead of 1 -norm, β, which can be written as -norm is used for the sparsity constraint to cacuate a sparse representation h β arg min β N β y β, (3) where the superscript is omitted for simpicity. (3) can be soved in a cosed-form soution by ridge regression [9], which is written as β T 1 T ( N N I ) N y, (4) and then foowed by x N h β P y, (5) where x denotes the SR resut and directy onto x. Note that of the time compexity. In summary, ANR earns off-ine regressors for fast SR, whie improving quaities using the neighborhood concept...4. Adusted ANR Method P is the proection matrix which proects an input LR patch P can be pre-cacuated, which improves the agorithm a ot in terms Adusted ANR (A+) method [5] is an improved ANR agorithm, which is based on the foowing observation. First, the dictionary atoms are sparsey samped in the space, whereas the training poo of image patch sampes obtained in off-ine training is practicay near-infinite. Second, the 49

4 oca manifod around an atom is spanned better by dense training sampes than by the dictionary atoms. Based on the observation, A+ method reformuates the optimization probem (3), which can be written as δ arg min δ S δ y δ, (6) where S denotes a LR dictionary, which contains K neighboring training sampes (possiby precacuated per atom in off-ine training). S repaces N that contains neighboring atoms in (3). As A+ uses the same baseine of ANR method, (6) can aso be soved by ridge regression. Thus, the cosed-form soution can be obtained as x S h δ ˆ P y, (7) where S h denotes HR dictionary corresponding to S and Pˆ represents the proection matrix obtained from S. Note that S h proects an input LR patch y directy onto HR patch x. Using such strategy, A+ method chooses better neighborhood for oca dictionaries, which drasticay improves the SR resut. 3. PROPOSED JOINT SR METHOD The proposed -based SR method generates image grid by referencing neighboring frames. The overa process of the -method SR is iustrated in Figure 1. The resoution of image grid is N times higher than that of the origina image in both width and height. N is reated to the sub-pixe accuracy for motion search. If N is set to 4, quarter-pe accuracy motion estimation is performed on the neighboring reference frames. A pixes in image grid are reconstructed using the information of neighboring reference frames. The reconstructed image grid has fu HR to reconstruct an HR image using down-samping. The down-samping process uses a super samping anti-aiasing (SSAA) method, which performs patterned samping method such as grid, random, Poisson, itter, and rotated grid. Rotated grid is we known as good for removing edges [10], therefore we use it for image samping. 50

5 Figure 1. An exampe of the proposed -based SR method. The sub-pixe accuracy is set to four. Circes denote the integer-pes and diamonds represent the sub-pes. Figure shows experimenta resuts of the proposed -based SR method according to different parameter setting. We adust three parameters, which are bock sizes, sub-pixe accuracy, and the number of reference frames. Defaut parameter setting is as foows: bock size = 3 3, subpixe accuracy = 1/, and the number of reference frames = 3. As shown in Figures (a)-(c), sma bock size reduces bock artifact in obect boundaries, however, computationa burden is increased. Figures (d)-(f) show that high eve of sub-pixe accuracy increases the quaity of the reconstructed HR image, however, a ot of hoes are produced (see Figure (f)) because defaut parameter setting for the number of references is too sma. To sove this probem, a arge number of references are used to obtain ots of tempora information (observed from Figures (g)-(i)), however, irreevant information can be presented by scene change. Figure (i) shows the quaity oss using irreevant reference frames in case of fast motion. In consideration of both image quaity and computationa compexity, we use the foowing parameter setting: bock size = 8 8, sub-pixe accuracy = 1/8, and the number of reference frames =

6 Figure. Experimenta resuts of the proposed -based SR method according to different parameter setting for the Stefan image sequence. Adusted ANR method [5] is used for exampe-based SR. After reconstructing an HR image using both - and exampe-based methods, the proposed method combines the two HR images [11] by using a gradient-based weight function. The fina reconstructed image is cacuated by ω ) F i, ω F i (8) F J ), where F J represents the fina reconstructed image, F denotes the HR image reconstructed by the method, and F is the HR image reconstructed by the exampe-based method. The weight functions are defined as ω ω g F g F ) ) (9) g F g F ) g F ) g F ) (10) ) 5

7 where g represents gradient operation and denotes convoution operation. The oint SR method enhances the reconstructed HR image quaity by weighting two images reconstructed by different approaches. 4. PERIMENTAL RESULTS AND DISCUSSIONS The simuated image sequence is a common intermediate format (CIF) video, of which spatia resoution equas at 30 frames per second. The test sequence named Stefan contains dynamic scenes paying tennis. One can observe cuttered background ocated at the top of the scene caused by crowd, fast motions of the tennis payers, arge camera motion tracking of the payers, and characters of various sizes on the was. In this paper, the CIF videos are down-samped to quarter CIF (QCIF) size, which equas SR is performed on the created QCIF videos to recover CIF videos. For exampe-based SR, the number of the atoms in the dictionary K is set to 16. For registration-based SR, sub-pixe accuracy N is set to 8, with the search range=3, and the number of reference frames=11. Figure 3. Experimenta resuts and performance comparison in terms of the PSNR for the LR image sequence (Stefan). Figure 3 shows the SR resut on the Stefan sequence with their cropped/zoomed images at the corners. For a reference of the comparison, SR resut using bi-cubic interpoation is shown in Figure 3(b). Exampe-based SR resut is shown in Figure 3(c). It can be observed that exampebased SR gives fine detais on the compex textures such as crowd. Aso, the resut by exampe- 53

8 based SR shows the best resut on the strong edges (see shouder of the tennis payer, horizonta ines across the image, and so on). However, Engish characters ocated at the bottom eft are not ceary readabe, since there is not sufficient information to recover the characters in a singe LR image. The -based SR method coects information from a number of adacent frames to buid an HR image, even though there is ony itte evidence in a singe frame. Figure 3(d) shows the SR resut by the proposed oint SR method which takes advantages from both - and exampebased approaches athough the peak signa to noise ratio (PSNR) is somewhat decreased. However, it shows better SR quaity on characters (shown in Figures 4(a) and 4(b)). Figure 4. Quaity comparison of the proposed oint SR with the A+ for three different image sequences. 54

9 Figures 4(a)-4(b) compare the proposed oint SR with A+ method for the Stefan sequence, whereas Figures 4(c)-4(d) for the Foreman sequence. The resut of the proposed oint SR method shows ess noise and higher contrast than A+ method. Figures 4(e) and 4(f) show the experimenta resuts performed on the fu HD-size ( ) Kimono image sequence. The proposed oint SR method up-scaes the Kimono sequence from fu HD to UHD 4K ( ) resoution. In contrast with the Stefan sequence (LR sequence), the experimenta resuts of the Kimono sequence (HR sequence) shows itte difference in quaitative comparison. 5. CONCLUSION The proposed SR method is focused on reconstructing HR video sequences from LR video sequences. The proposed -based SR method successfuy coects information from adacent frames to reconstruct Engish characters in the video sequences. However, the exampe-based SR method gives better textures and strong edges in the resut HR video. In this paper, - and exampe-based SR methods are fused based on the gradient features. The proposed oint SR method gives smaer PSNRs than the exampe-based method, however it shows better reconstruction resuts on high-eve features. Future work wi focus on optimizing the oint SR method using convoutiona neura network to reduce the time compexity of the agorithm. 6. ACKNOWLEDGMENTS This work was supported in part by the Brain Korea 1 Pus. REFERENCES [1] Y. Tian and K.-H. Yap, Joint image registration and super-resoution from ow-resoution images with zooming motion, IEEE Trans. Circuits and Systems for Video Technoogy, vo. 3, no. 7, pp , Ju [] C. Liu and D. Sun, On Bayesian adaptive video super resoution, IEEE Trans. Pattern Anaysis and Machine Inteigence, vo. 36, no., pp , Feb [3] Z. Xiong, D. Xu, X. Sun, and F. Wu, Exampe-based super-resoution with soft information and decision, IEEE Trans. Mutimedia, vo. 15, no. 6, pp , May 013. [4] R. Timofte, V. D. Smet, and L. V. Goo, Anchored neighborhood regression for fast exampe-based super-resoution, in Proc. IEEE Int. Conf. Computer Vision, pp , Sydney, Austraia, Dec [5] R. Timofte, V. D. Smet, and L. V. Goo, A+: Adusted anchored neighborhood regression for fast super-resoution, in Proc. Asian Conf. Computer Vision, pp. 1 15, Singapore, Nov [6] R. Timofte and L. V. Goo, Adaptive and weighted coaborative representations for image cassification, Pattern Recognition Letters, vo. 43, pp , Ju [7] Y. Zhu, Y. Zhang, and A. L. Yuie, Singe image super-resoution using deformabe patches, in Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp , Coumbus, OH, USA, Jun [8] R. Zeyde, M. Ead, and M. Protter, On singe image scae-up using sparse-representations, in Lecture Notes in Computer Science: Curves and Surfaces, J.-D. Boissonnat and P. Chenin, Eds., Springer, pp , 01. [9] Y. Zhu, Y. Zhang, and A. L. Yuie, Singe image super-resoution using deformabe patches, in Proc. IEEE Computer Vision and Pattern Recognition, pp , Coumbus, OH, USA, Jun [10] R. Barringer and T. A. Moer, A4: Asynchronous adaptive anti-aiasing using shared memory, ACM Trans. Graphics, vo. 3, no. 4, pp. 100:1 100:10, Ju [11] K. Lee and C. Lee, High quaity spatiay registered vertica tempora fitering for deinteracing, IEEE Trans. Consumer Eectronics, vo. 59, no. 1, pp , Feb