Improvement of Spatal Resoluton Usng BlockMatchng Based Moton Estmaton and Frame Integraton Danya Suga and Takayuk Hamamoto Graduate School of Engneerng, Tokyo Unversty of Scence, 6-3-1, Nuku, Katsuska-ku, Tokyo, 125-8585, Japan Emal: {suga, hamamoto}@sl.ee.kagu.tus.ac.p resoluton va the dgtal post-processng s beng actvely studed [1]. Because hgh frame-rate mages have a hgh temporal correlaton, the amount of moton between frames s very small. Ths characterstc has several advantages n the moton estmaton appled to magng systems for reconstructng hgh spatal resoluton mages. For example, t mproves the accuracy of gradent-based moton estmaton because t can effectvely estmate small motons, but cannot cope wth wde-range moton [2], [3]. In addton, the amount of calculatons n the block matchng-based method s reduced as ts search area becomes smaller. Hgh accuracy moton estmaton technques based on the gradent method that explots hgh frame-rate mages have already been proposed n our prevous work [4], [5]. The technques proposed can reduce the estmaton error n the spatal gradent usng the hgh temporal correlaton characterstc of the hgh frame-rate mages. Ths paper proposes a new spatal alasng resstant moton estmaton method that mproves spatal resoluton for reconstructon. The results of moton estmaton accuracy and mprovements n the spatal resoluton of hgh framerates mages obtaned usng the proposed method are also presented and evaluated. Abstract Hgh frame-rate mages typcally possess a hgh temporal correlaton between ther moton vectors. By use of ths characterstc, hgh accuracy moton estmaton technques for hgh frame-rate mages were proposed n our prevous work. Compared wth conventonal methods, our proposed methods are relatvely resstant to spatal alasng errors. However these methods are stll nadequate for resstant to spatal alasng errors. Ths paper proposes a new matchng based moton estmaton method that further mproves spatal resoluton. The proposed moton estmaton method s more resstant to spatal alasng errors and s sutable for reconstructng a hgh spatal resoluton mage usng multple hgh frame-rate mages. The method uses multple prevous frames for moton estmaton. Frst, moton vectors are ndependently estmated between the current frame and multple prevous frames usng the basc block matchng method and parabola fttng. Next, a tmeaxally-averaged moton vector s approxmated from multple moton vectors usng the least-squares method. Fnally, the moton vector n the current frame s calculated by the approxmaton straght lne. The results of moton estmaton experments conducted usng mages wth spatal alasng errors show that the proposed method mproves accuracy. Further, the results of super-resoluton experments conducted usng hgh frame-rate mages show that the proposed method also mproves spatal resoluton. Index Terms hgh frame-rate, moton estmaton, super resoluton, alasng error II. I. INTRODUCTION A. Block Matchng Method The block matchng method s a way of moton estmaton. Ths method compares a target macroblock n the prevous frame wth neghborhoods of macroblocks n the current frame, and determnes the best matched coordnates by usng an evaluaton functon. One of the popular functons s the sum of squared dfferences (SSD). The relaton between these quanttes can be expressed as: Image sensng devces play an mportant role n mage processng systems, because they determne the qualty of the nput mage. Hence, these devces are contnually beng mproved n order to obtan hgh-qualty mages. Further, ther characterstcs, such as hgher senstvty, lower nose, and hgh spatal resoluton, are constantly beng upgraded. In fact, to acheve hgh frame-rates, low power consumpton or downszng, many researchers have consdered on-sensor analog-to-dgtal converters. However, t s dffcult to mprove spatal resoluton effectvely such a devce owng to the resoluton lmts of optcal lens. Therefore, the mprovement of spatal MOTION ESTIMATION BASED ON THE MATCHING METHOD S sdd f, g dx, dy 2 where, are pxel postons. f(, ) s the pxel value of a target macroblock n the prevous frame, g(, ) s the pxel value of macroblock n the current frame, and dx and dy denote the moton n the and drectons, respectvely. Manuscrpt receved Aprl 5, 2015; revsed August 31, 2015. do: 10.18178/og.3.2.117-121 (1) 117
Some methods have been proposed for sub-pxel moton estmaton based on the block matchng algorthm. One method nterpolates sub-pxel values by spatal flterng processng. After expandng mage sze by a spatal flter, sub-pxel moton vectors are estmated usng the block matchng method. Another method uses parabola fttng over three ponts [6]. The estmated subpxel dsplacement dˆ by parabola fttng can be expressed as: ˆ R( 1) R(1) d (2) 2 R( 1) 4 R(0) 2 R(1) where R(s) s a smlarty functon and s s the ntegral shfted value from the extremal pxel poston. Further, the range of dˆ s usually defned as 0.5 d ˆ 0. 5. Usng the SSD, R x (s x ) and R y (s y ) are expressed as: x s,, 2 x f g dx sx dy R (3) y s,, 2 y f g dx dy sy R (4) B. Proposed Method Before estmatng moton vectors, we frst degrade the hgh spatal frequency by a low-pass flterng. Ths process s carred out to reduce random noses or alasng effects, and mprove the accuracy of moton estmaton. However, when an acqured mage has alasng errors, the shapes of the acqured mage are dfferent from those of the real mage, and t s dffcult to correctly estmate moton vectors usng the acqured mage. Fg. 1 llustrates ths stuaton. It shows how an acqured mage loses the shape of the real mage owng to a lower spatal resoluton: a lower spatal resoluton means that the pxel ptch s long. If the real mage moves wth sub-pxel precson, the shape of the acqured mage wll change as a result of alasng effects. Ths results n dffculty estmatng the moton vectors wth sub-pxel precson. In other words, as shown n Fg. 2, f the moton vectors are known, the shape of the real mage can be reconstructed usng several frames. To reduce errors due to alasng effects, we propose the block-matchng based moton estmaton. Ths moton estmaton uses multple prevous frames nstead of only one prevous frame. Fg. 3 and Fg. 4 ndcate the concept underlyng the proposed method. Frst, moton vectors are ndependently estmated between the current frame and multple prevous frames, usng the basc block matchng method and parabola fttng. Further, approxmaton straght lnes n the x and y drectons are calculated from the estmated moton vectors usng the least-squares method. Tme-axally-averaged current moton vectors are then derved from the calculated approxmaton straght lnes. Ths method s based on the concept that hgh frame-rate mages have hgh temporal correlaton and the moton vectors are regarded as havng unform moton. The characterstc of ths proposed method reduces errors due to alasng effects and varous noses. Fgure 1. Acqured mage wth error due to alasng effects. Fgure 2. A real mage reconstructed from several frames. Fgure 3. Concept underlyng the proposed moton estmaton method usng multple prevous frames. Fgure 4. Proposed moton estmaton method usng an approxmaton straght lne. 118
moton amount from the basc mage, and the vertcal axs ndcates the average error of the estmated moton. The followng fve methods (nclusve of the proposed method) were compared: (1) The block matchng and parabola fttng (α=1) (2) The two tmes expanded block matchng and parabola fttng (α=2) (3) The general gradent method (4) The varable gradent method (5) The proposed method The block sze M for estmatng the moton was 21 21. In block matchng based methods (1) and (2), the search area N was set to be 3 3, wth α ndcatng the expanson rato of the block matchng process. In proposed method (5), the search area N was also set to 3 3. Moton estmaton was calculated usng 10 frames. In Fg. 6, the accuracy of the moton estmaton usng the proposed method s the same as those obtaned usng the two tmes expanded block matchng and parabola fttng, and the varable gradent method. In Fg. 6, the average errors of the moton estmaton usng block matchng based methods (1) and (2) fluctuate sgnfcantly. The errors obtaned usng gradent based methods (3) and (4) have a lot. By contrast, the accuracy of the moton estmaton usng the proposed method, (5), s the hghest on average. These results show that the proposed method s hghly resstant to spatal alasng noses. Fgure 5. 1/8 pxel shfted mages: Images wth low alasng noses. Images wth alasng noses. B. Evaluaton of Super Resoluton The super-resoluton algorthm based on the ntegraton of hgh frame-rate mages s descrbed n ths secton. As shown n Fg. 7, one hgh-resoluton mage s reconstructed usng multple mages. Frst, regstraton s performed: The moton vectors are estmated from each of the reference frames to the target frame and pxel values are gathered to the target frame from reference frames based on the estmated vectors. Next, reconstructon s performed: The target frame wth the pxel values of sub-pxel postons s fltered only once to reconstruct a hgh resoluton mage. Fgure 6. Evaluaton of average error obtaned by the varous moton estmaton methods: Evaluaton results for Fg. 5. Evaluaton results for Fg. 5. III. EVALUATION A. Evaluaton of Moton Estmaton The expermental results for the average error of the varous moton estmaton methods usng syntheszed mages are shown n ths secton. The evaluaton used mages that were each horzontally shfted by each 1/8 pxel, as shown n Fg. 5 and Fg. 5. Fg. 5 has more alasng noses than Fg. 5, and Gaussan nose at approxmately PSNR 34dB was added to these mages. The results are shown n Fg. 6 and Fg. 6. In Fg. 6 and Fg. 6, the horzontal axs ndcates the Fgure 7. Flow of ncreasng spatal resoluton. In ths experment, a 384 512 mage was reconstructed usng nne 192 256 mages. Fg. 8 shows the orgnal 119
mage obtaned at 300 fps. Ths mage has alasng errors, and the moton amount s approxmately 0.25 pxel/frame. We used four of the moton estmaton methods lsted n the prevous secton (the block matchng and parabola fttng (α=1) method was excluded). The bcubc nterpolaton algorthm was used to carry out the flterng for reconstructon. usng the proposed method (e) s slghtly hgher than that obtaned usng the gradent method (c). The results ndcate that the accuracy of the super-resoluton s mproved by applyng the proposed moton estmaton method. The results of ths experment show that the proposed method s effectve as a super-resoluton algorthm for actual mages that nclude spatal alasng errors at a hgh frame-rate. IV. (c) (d) CONCLUSION Ths paper proposed a new moton estmaton method based on block matchng. The proposed method uses multple frames and approxmates moton vectors va a lne to engender hgh resstant to spatal alasng errors. The results of the experment conducted usng mages wth low alasng errors showed that the accuracy of the moton estmaton by the proposed method s the same as that of prevous methods. However, the results of the experment conducted usng mages wth alasng errors show that the accuracy of the moton estmaton by the proposed method s hgh on average. In addton, applyng the proposed method to the super-resoluton algorthm, the expermental results ndcate that the proposed method mproved the spatal resoluton of hgh frame-rate mages wth spatal alasng errors. In future work, we wll nvestgate not only the moton estmaton method but also the super-resoluton algorthm havng resstant to spatal alasng noses. REFERENCES [1] [2] [3] (e) Fgure 8. Expermental results: Orgnal mage. Block matchng and parabola fttng (α=2). (c) Gradent method. (d) Varable gradent method. (e) Proposed method. TABLE I. [4] [5] RESULTS OF PSNR Fgure number PSNR (db) 8 8(c) 24.95 24.90 8(d) 25.94 8(e) 26.26 [6] Danya Suga receved the B.E. and the M.E. n electrcal engneerng from Tokyo Unversty of Scence n 2001, and 2003. He s currently a Ph.D. student at the Graduate School of Engneerng, Tokyo Unversty of Scence. Hs research nterests are mage processng and mage sensors. Fgs. 8-Fg. 8(e) show the expermental results obtaned usng the super-resoluton algorthm. The PSNR for each method s gven n Table I. The results of PSNR usng block matchng and parabola fttng and gradent method (c) are almost the same at approxmately 25dB. The results of PSNR usng the varable gradent method (d) and the proposed method (e) are better than the above methods at approxmately 26dB. The PSNR S. C. Park, M. K. Park, and M. G. Kang, Super-Resoluton mage reconstructon: A techncal overvew, IEEE Sgnal Processng Magazne, vol. 20, no. 3, pp. 21-36, 2003. S. Lm, J. G. Apostolopoulos, and A. E. Gamal, Optcal flow estmaton usng temporally oversampled vdeo, IEEE Transactons on Image Processng, vol. 14, no. 8, pp. 1074-1087, 2005. R. Battt, E. Amald, and C. Koch, Computng optcal flow across multple scale: An adaptve coarse-to-fne strategy, Internatonal Journal of Computer Vson, vol. 6, no. 2, pp. 133145, 1995. D. Suga, H. Katayama, and T. Hamamoto, Image emphass based on ntegraton of hgh frame-rate mages by new gradentbased moton estmaton method, n Proc. IEEE Internatonal Symposum on ISPACS, 2012, pp. 118-123. Y. Sato, D. Suga, and T. Hamamoto, Gradent-Based moton estmaton usng characterstc of hgh frame-rate mages, n Proc. Internatonal Workshop on Advanced Image Technology, 2013, pp. 23-28. M. Shmzu and M. Okutom, Precse sub-pxel estmaton on area-based matchng, n Proc. Eghth IEEE Internatonal Conference on Computer Vson, 2001, pp. 90-97. 120
Takayuk Hamamoto receved the B.E. and the M.E. n Electrcal Engneerng from Tokyo Unversty of Scence n 1992, and 1994, respectvely, and the Dr.E. n Electrcal Engneerng from the Unversty of Tokyo, n 1997. Currently, he s a Professor at the department of electrcal engneerng of Tokyo Unversty of Scence Hs current research nterests are mage processng and computatonal mage sensors. 121