Motion Compensated Frame Interpolation Using Motion Vector Refinement with Low Complexity

Transcription

1 Motion Compensated Frame Interpolation Using Motion Vector Refinement with Low Complexity Jun-Geon Kim and Daeho Lee Abstract-- In this paper, we propose a noel method of motion compensated frame interpolation (MCFI). We use the bilateral motion estimation (BME) scheme haing a lower computation to estimate motion ectors (MVs) for an interpolated frame. By applying ector refinement, precise MVs can be calculated to enhance qualities of complex images. Finally, Oerlapped block motion compensation (OBMC) is applied to reduce blocking artifact. Experimental results show that our method offers the faster computation speed and image improement in complex scene. Index Terms Bilateral motion estimation, frame rate up-conersion, motion compensated frame interpolation, ector refinement I. INTRODUCTION The frame rate up-conersion (FRUC) is technique to increase a temporal resolution by generating an intermediate frame by using adjacent frames. It is important area to hae a ariety of application. In ideo transmission, temporal down-sampling is frequently used on channel haing a bandwidth restriction. Howeer, although a periodic skipping of frame is used, it necessarily causes to degrade the smooth motion quality of ideo. So, it needs temporal up-sampling by FRUC scheme in receier unit to hae a good quality. Another application is motion blur reduction for liquid crystal display (LCD) teleision. LCD usually shows the phenomenon near fast moing object, when it has low frame rates. The motion blur fundamentally arises from the slow response time and hold-type rendering of LCD. FRUC scheme becomes one solution, which generates more frames such that the LCDs can display these frames faster, in this situation. [, ] FRUC algorithms may be roughly diided in two methods by whether they use motion information or not [3, 4]. The latter utilizes simple methods such as linear interpolation and frame repetition etc. It is simple to implement in This research was funded by the MSIP(Ministry of Science, ICT & Future Planning), Korea in the ICT R&D Program 03. Jun-Geon Kim is now with the Department of Electronics and Radio Engineering, Kyung Hee Uniersity, Yongin-si, Gyeonggi-do, Korea. ( joongoen@khu.ac.kr ). Daeho Lee is an Associate Professor in the Humanitas College, Kyung Hee Uniersity, Yongin-si, Gyeonggi-do, Korea. ( nize@khu.ac.kr ). ISBN: hardware or software, but it incurs jerkiness and ghost phenomenon for moing object. In case of the former, because of constructing an interpolated frame by using motion of object, it is referred to as motion-compensated frame rate up-conersion (MC-FRUC). This method has more complex than the preious one, but it is widely used because of generating a high quality interpolated frame. MC- FRUC is composed of two steps: first, motion estimation (ME) and second, motion-compensated frame interpolation (MCFI). In ME step, block matching algorithm (BMA), which calculates similarities between blocks by using sum of absolute differences of two blocks pixel alue, is commonly used to find a motion ector field of the entire frame being predicted. And then, an intermediate frame is generated by interpolating each block according to corresponsie motion ector (MV). In preious work, seeral techniques to discoer MVs are proposed. Forward ME, one among them, is that a block in current frame compares with one in search range of preious frame considered as a reference frame to find MVs. Backward ME makes a current frame as a reference frame. And, Bilateral ME (BME) [5] and BME with side matching distortion (SMD) [6] are proposed. The scheme considers similarity between predicted block and already interpolated surrounding blocks to reduce block artifacts, but it has highly complex computation and also, high quality is not guaranteed for complex scenes. Because the performance of FRUC algorithms strongly depends on estimated MV and interpolation process, it is proposed that MV is smoothed with its neighboring MVs to get more accuracy MVs [7]. In this paper, we apply ector refinement to BME with SMD, whereby MVs predicted is refined by nearby MVs for much accurate it. Consequently, it leads to generation of quality frame in seeral sequences. It generally shows good performance through seeral experiment sequences in terms of both objectie and subjectie ealuation. Especially, it has adantage of haing relatiely better performance in complex or translational sequence. So, we sole the problem that computation of bilateral ME with SMD is ery high by lowering unnecessary algorithm repeat number. II. MOTION COMPENSATED INTERPOLATION A. Bilateral ME with SMD haing complexity reduction References [5, 6] proposed bilateral ME that find the motion ectors of intermediate frame. This scheme supposes that motion trajectory passes through intermediate frame linearly from a block in preious frame to one in next frame. Specifically, we acquire a motion trajectory by comparing a

2 block in preious frame and one in next frame which are at symmetric position oer one in preious frame. Fig. depicts an aboe situation well. Let f n-, f n and fn+ denote a preious, intermediate and next frame, respectiely. Let s denotes a position indicating a pixel of a block in intermediate frame f n and denotes one of candidates of ectors of current block. And then, one pixel on s in an intermediate frame is mapped to one pixel on s - in preious frame and one pixel Fig.. Bilateral motion estimation. on s + in the next frame. We determine a MV among MV candidates concerning a block by comparing only all blocks in symmetric relation through the cost function as SBAD calculated by SBAD [B, ] = f [ ] f [ s+ ]. () i, j n- n+ s B After SBAD is computed in all each candidate motion ector, a motion ector that minimizes SBAD alue is chosen as MV of current block. Howeer, it doesn t produce a reliable motion ector when objects hae rotational or zooming motion. So, Bilateral ME with SMD is proposed. This algorithm adds side match distortion that measures a similarity between neighboring blocks and a predicted block by using side pixel alues of a current block and neighboring blocks as shown in Fig.. To judge a similarity, it computes the aerage of sum of absolute pixel differences between borders of neighboring and a predicted block by N SMD[B i,j, ] = fˆ [g ˆ n k, ] f n[h k], () N k= 0 where gk and h k mean the position of the kth pixel on four boundaries of block in the predicted block and neighboring block, respectiely. And, N and fˆ n [g k, ] denotes the number of border pixel in predicted block and an interpolated pixel alue in an intermediate frame when a motion ector is set to. Keep in mind an interpolated pixel, f ˆ [h ] n k, in neighboring blocks should be usable when SMD is computed. Therefore, there are no pixel alues of any neighbor blocks when it, firstly, is calculated in a leftmost and uppermost block. There is only a left neighboring block in second block. There are, except for blocks on first column, both left and upper block from second row until last row. Once entire interpolated blocks are obtained, there are interpolated left, right, upper and under block as shown in Fig.. We select the best motion ector i,j, which minimizes a weighted sum of the SBAD and the SMD, for block, B i,j, being processed in intermediate frame. i,j = arg min{ u SBAD [B i,j, ] + ( u) SMD [B i,j, ] }, (3) Fig.. Side matching distortion. where u is a weighting factor, 0< u <. As referred to earlier, since an under and right block is not interpolated yet in the first processing step, SMD is calculated by only left and upper one in this step. After that, the estimation process utilizing four blocks is iteratiely used until all motion ector estimated is not changed. We experimented how the change of iteration number affects a subjectie and objectie performance of algorithm and to compare computation speed according to iteration number. And, we found the proper number of iteration.

3 B. Vector refinement The quality of motion-compensated frame highly depends on how precisely motion ector is estimated. Thus, obtaining precise motion ectors is ery important. We use the ector refinement method, which is proposed in [5], for precise motion ector to be extracted. Refinement procedure of the algorithms is in followings; let ( X) denote motion ector of x block. SBADs of nine motion ector including ( B) and ( N ~8 ) are calculated and then, we determine the motion ector that minimizes SBAD by min ( B) = arg min SBAD [ B, ], (4) (B),(N ), 0 i 8. where { } i fˆ ( s) = [ ˆ ( s) + ˆ () s + ˆ () s + ˆ R f ()] s n, fn, fn,3 fn,4 4 In R region, fˆ ( s) = [ ˆ () s + ˆ R f ()] s n, fn,4, s R. (6), s R. (7) Process described aboe is applied to all two or four oerlapped region in entire intermediate frame. Original pixel f ˆ s, is used in non-oerlapping region. alue, ( ) n Fig. 4. Motion-compensated frame of garden when OBMC is not applied. Fig. 3. B is block that MV being refined belongs to. N ~ N 8 are neighboring blocks We can discoer that the performance of algorithm generally is improed by adding ector refinement technique to bilateral ME with SMD and especially, it shows better performance in sequences haing translational moement or complex pattern. C. Oerlapped block motion compensation If we try to generate motion-compensated frame without OBMC, it necessarily results in ugly intermediate frame due to blocking artifacts as shown in red box of Fig. 4. Thus, OBMC is needed to reduce the artifacts. So, we select the method proposed by [5], because of simplicity. Key idea of the OBMC is smoothing the boundaries between blocks by oerlapping pixels around boundary. The method for OBMC is described well in Fig. 5 and detailed calculation process of it is as in the followings; n, s fn- s fn+ n, s fn- s fn+ n,3 s fn- s fn+ n,4 s fn- s fn+ where B denotes extended block. E,x In R region, E,, (5a) E,, (5b) E,3, (5c) E,4, (5d) Fig. 5. OBMC is depicted. Let O,E and w denote original block size, extended block size and extension width. R and R means four oerlapping region and two oerlapping region, repectiely. III. EXPERIMENTAL RESULTS We temporally reduce the frame per second (fps) of experimental sequences by factor of two for comparing analysis. Specifically, een frames constructed by MCFI are compared by an original one in the objectie and subjectie iew. Experimental sequences are as in the followings; The CIF sequences are: Coastguard, Container, Forman and Hall monitor and the SIF sequences are: Football, Garden, Mobile and Tennis. We set a block size to 5 pixels and search range to pixels for both horizontal and ertical directions. And, extension width, w, is set to 5. We found out that effect of ector refinement and its excellence by comparing two

4 algorithms which are BME with SMD + OBMC and BME with SMD + ector refinement + OBMC. And we found that the conergence of BME with SMD algorithm is not always ensured in all sequence. Also, we experimented how the change of iteration number of the algorithm affects subjectie and objectie performance. Computation speed of (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) Fig. 5. Comparison graphs of PSNR according to sequences and subjectie ealuation by comparing constructed frame on proposed method with one on the other. (a) football, (b) garden, (c) mobile, (d) tennis, (e) coatguard, (f) container, (g) foreman, (h) hall monitor, (i) 8 frame of Garden sequence, (j) the frame generated by BME with SMD + OBMC, (k) the frame generated by proposed. TABLE I COMPARISON OF THE AVERAGE PSNR AND PROCESSING TIMES ACCORDING TO REPETITION NUMBER OF ALGORITHM Sequences Football Tennis Foreman Repetition Number PSNR Processing Times PSNR Processing Times PSNR Processing Times Prop. other Prop. other Prop. other Prop. other Prop. other Prop. other The unit of processing time is millisecond. the algorithm according to iteration number is measured to determine the proper number of iteration. As shown in Fig. 5, when ector refinement is applied, PSNR is generally increased in all eight sequences. Among sequences, PSNR differences in Garden and Container are

5 remarkably bigger than the other sequences. And then, we can conclude that its performance is maximized on frame haing complex shapes or translational moement. In addition, it shows isual betterment in subjectie ealuation, as shown in (k) of Fig. 5. It is said that BME with SMD conducts iteratie process until MV is not changed in [6]. Howeer, we discoered the fact that this condition is not satisfied on a certain frame in football or tennis. And then, we try to understand relatiity between repetition number and PSNR. Consequently, repetition number seldom affect performance in more than 0 or 0 case as shown in Table I and rather, reduce processing speed. As result by obseration, the salient part resulting in slow speed is precisely the frame that doesn t hae conergence. Thus, we properly set repetition number as ten. All graphs in Fig. 5 are also experimented by ten repetition number. IV. CONCLUSION We proposed the MCFI using ector refinement method. The proposed method generally shows improement of performance in all sequences and especially, shows the better results in complex sequence like a garden or sequence haing a translational motion. Also, it makes existing bilateral ME technique haing low processing speed becomes rapid by reducing an unnecessary algorithm repetition. Because OBMC is used for simplicity in this paper, if improed one is applied to our scheme, a better isual result is anticipated In future work, we will add an ability distinguishing whether it is edge of image or not to the existing bilateral ME with SMD to make a new scheme of better performance V. REFERENCES [] N. Jacobson, Y.-L. Lee, V. Mahadean, N. Vasconcelos, and Truong Q. Nguyen, A Noel Approach to FRUC Using Discriminant Saliency and Frame Segment, IEEE Trans. Image Process., ol. 9, no., pp , No. 00. [] C. Wang, L. Zhang, Y. He, and Y.-P. Tan, Frame Rate Up-Conersion Using Trilateral Filtering, IEEE Trans. Circuits Syst. Video Technol., ol. 0, no. 6, pp , Jun. 00 [3] S.-J. Kang, K.-R. Cho, and Y. H. Kim, Motion compensated frame rate up-conersion using extended bilateral motion estimation, IEEE Trans. Consum. Electron., ol. 53, no. 4, pp , No [4] R. Han and A. Men, Frame Rate Up-Conersion for High-Definition Video Applications, IEEE Trans. Consum. Electron., ol. 59, no., pp. 9-36, Feb. 03. [5] B.-T. Choi, S.-H. Lee, and S.-J. Ko, New frame rate up-conersion using bi-directional motion estimation, IEEE Trans. Consum. Electron., ol. 46, no. 3, pp , Aug [6] B.-D. Choi, J.-W. Han, C.-S. Kim, and S.-J. Ko, Motion-compensated frame interpolation using bilateral motion estimation and adaptie oerlapped block motion compensation, IEEE Trans. Circuits Syst. Video Technol., ol. 7, no. 4, pp , April 007. [7] A. H. Huang and T. Q. Nguyen, A multistage motion ector processing method for motion-compensated frame interpolation, IEEE Trans. Image Process., ol. 7, no. 5, pp , May 008. [8] M. T. Orchard and G. J. Sullian, Oerlapped block motion compensation: An estimation-theoretic approach, IEEE Trans. Image Process., ol. 3, no. 5, pp , Sep [9] S. Dikbas and Y. Altunbasak, Noel True-Motion Estimation Algorithm and Its Application to Motion-Compensated Temporal Frame Interpolation, IEEE Trans. Image Process., ol., no. 8, pp , Aug. 03.