Advanced De-Interlacing techniques with the use of Zonal Based Algorithms

Transcription

1 Advanced De-Interlacing techniques with the use of Zonal Based Algorithms Alexis M. Tourapis 1, Oscar C. Au 2, Ming L. Liou Department of Electrical and Electronic Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong ABSTRACT This paper describes a new highly efficient deinterlacing approach based on motion estimation and compensation techniques. The proposed technique mainly benefits from the motion vector properties of zonal based algorithms, such as the Advanced Predictive Diamond Zonal Search (APDZS) and the Predictive Motion Vector Field Adaptive Search Technique (PMVFAST), multihypothesis motion compensation, but also an additional motion classification phase where, depending on the motion of a pixel, additional spatial and temporal information is also considered to further improve performance. Extensive simulations demonstrate the efficacy of these algorithms, especially when compared to standard deinterlacing techniques such as the line doubling and line averaging algorithms. Keywords: Deinterlacing, motion estimation, zonal algorithms, PMVFAST, transcoding, standards conversion, video stills, video signal processing, temporal interpolation. 1. INTRODUCTION Frame N Frame N+1 Frame N Top Field Frame N+1 Bottom Field Vertical Vertical Horizontal Horizontal Temporal Temporal Figure 1: Illustration of Progressive Scan versus Interlace Scan Video Interlaced scan has been an essential part of TV and Video Coding Schemes for several years in order to address three conflicting factors, bandwidth, flicker, and resolution. A frame is basically split into two fields, one consisting of even indexed lines named the even field and a second consisting of odd indexed lines named the odd field (Figure 1). These fields are sampled at different time instants. This allows us to increase, or essentially double the temporal sampling rate while at the same time keeping the total number of pixels per second unchanged. Unfortunately interlaced scan also introduces several other visual artifacts known as crawl, twitter, and flickering [1]. In addition, it makes little sense in displaying or even printing a single frame out of a sequence, whereas, in some cases, it is necessary to display interlaced video on a 1 Correspondence: alexismt@ieee.org 2 Correspondence: eeau@ee.ust.hk; Telephone (+852) , Fax (+852)

2 progressive monitor. For these and many other reasons it is quite important to be able to convert interlaced to progressive video, while at the same time providing improved vertical resolution in the vertical direction. Several interpolation schemes have been previously presented in order to solve this problem. The idea behind these techniques is to essentially reconstruct the missing lines before having to display a frame. Such techniques include spatial methods [3-5], motion adaptive techniques [6], and approaches using motion compensation [7-9]. From these, spatial methods are usually the simplest and fastest to implement but tend to introduce more artifacts and could deteriorate image quality rapidly. From these techniques, Line doubling (LDB) and Line Averaging (LAV) [2] are quite popular for the display of interlaced video on personal computers. On the other side, motion compensated techniques have been shown to yield better results but significantly rely on robust and reliable motion models, which in comparison introduce excessive computational complexity. Recently new highly efficient motion estimation algorithms have been proposed, namely the zonal based algorithms [10-12], which could significantly reduce complexity, while at the same time, achieve similar quality as the Brute Force Full Search (FS) algorithm. These algorithms, such as for example the Predictive Motion Vector Field Adaptive Search Technique (PMVFAST) [10,11], mainly rely on the consideration of several predictors in the motion estimation process, such as the motion vectors of the adjacent blocks at left, top, and top right from the current block, their median value, the (0,0) motion vector, and the previous frame collocated block s motion vector. This, with the addition of other fuzzy logic techniques such as adaptive thresholding, and the pattern of the search used, enable these algorithms not only to perform as good as the FS algorithm in terms of quality, but also can achieve a motion field close to the true motion of the frame (Figure 2). As we have previously mentioned, this is quite important in the case of motion compensated techniques and suggests that these algorithms would be an excellent choice for deinterlacing. (a) (b) Figure 2: Motion field of frame 8 of Foreman using (a) FS, (b) a Zonal Based algorithm. In this paper we present a new advanced motion compensated technique for deinterlacing. Unlike other previous methods, the presented technique tries to generate the missing fields by using a multihypothesis motion compensation approach. In addition, a motion classification process is also introduced according to which, if motion for particular areas satisfies some particular conditions, additional processing, for example spatial or temporal filtering, is also performed. This results in significantly improved performance versus simple motion compensated techniques. Even though this approach mainly benefits from the smooth motion vector field that zonal algorithms generate, we further demonstrate that the technique can perform very well even when other motion estimation algorithms are used. In section 2 we briefly give the advantages of Zonal algorithms and how we can use such properties in deinterlacing. We also discuss how we can use a multihypothesis motion compensation approach to improve the quality of the predicted field. In section 3, we describe how motion is classified, and depending on the classification we introduce further additional steps in the deinterlacing process. Finally in section 4 extensive simulation results are presented which demonstrate the excellent performance of the proposed scheme.

3 2. ZONAL ALGORITHMS AND MULTIHYPOTHESIS MOTION COMPENSATION FOR DEINTERLACING As was discussed in the previous section, zonal based algorithms can achieve a smoother motion vector field, and relatively more accurate, and closer to the true motion, prediction of the motion vectors compared to other algorithms, including FS. This can be also observed from Table 1 where the entropy of the difference of the current motion vector versus other closely related motion vectors of spatially and temporally adjacent blocks is given. To understand these numbers, consider that adjacent blocks are highly correlated, and thus, in reality, tend to have also similar motion vectors. This would also suggest that, in most cases, the closer to the true motion field an algorithm gives, the smaller the entropy versus a set of possible predictions would be as well. These predictors include the motion vectors of the three spatially adjacent blocks on the left, top, and top right to the current position, their Median, the (0,0) motion vector, and even the motion vector of the collocated block in the previous frame. All these predictors constitute a Predictor Set (Set A), and are quite important in the performance of zonal algorithms. Obviously as we can see in Table 1 zonal algorithms have significantly lower entropy compared to other algorithms such as FS, and combined with Figure 2 we may claim that they give a rather reliable and close to the true motion, motion vector field. For the purpose of our experiments we have selected using the Predictive Motion Vector Field Adaptive Search Technique (PMVFAST) [10,11]. This algorithm can be essentially considered as a special case of the Zonal Algorithms. Other zonal algorithms such as the Advanced Predictive Diamond Zonal Search (APDZS) [12] could also be selected with similar or better performance. It should be noted that PMVFAST was recently accepted as a recommendation for motion estimation in the Optimization Model 1.0 of MPEG-4 [13] and it is characterized by its superior speed up and quality versus most if not all other algorithms. This is quite important in a practical implementation since, considering that for a specific sequence motion vectors could possible have been already estimated using PMVFAST, we may immediately use these motion vectors for deinterlacing, without having to repeat the entire motion estimation process. Full Search Zonal Algorithm Sequence (0,0) Median Set A (0,0) Median Set A Foreman Stefan Bus Table 1: Motion Vector Entropy versus different Predictors We first perform field motion estimation, using a block size of 16 8, using the fields of interest (i.e. the even fields) from the current and previous frames. By doing so we may generate the motion field for a time interval t. What though interests us is to make an estimate of the motion field at a time interval t/2. This is achieved by simply dividing all motion vectors by 2. Afterwards using motion compensation techniques we can generate the missing field lines. These missing fields may be then interleaved with the original fields at time interval t/2 (in this case the odd fields) in order to generate the progressive frame. Motion estimation using non-corresponding fields could also be used to further improve performance. C 1 (x+dx,y+dy) (x+dx/2,y+dy/2) C 3B MV (dx/2,dy/2) (x,y) B 3 B 1 MV (dx,dy) (x,y) (x-dx/2,y-dy/2) C3F MV (-dx/2,-dy/2) field in (k-1) th frame inserted field field in k th frame Figure 3: A simple MV interpolation method for predicting the missing field One question that may arise is what is the best method to perform motion compensation. Let us assume that we have performed motion estimation for the corresponding fields in the k th frame versus the (k-1) th frame. Let us assume that block B 1 at position (x,y) has a motion vector (dx,dy). The easiest method for constructing the missing field, but nevertheless a not

4 so accurate one, would be to assume that the block at the corresponding (x,y) position inside the missing field B 3 also has a motion vector equal this time to (dx/2,dy/2) pointing in the (k-1) th frame (block C 3B ). This can be seen in Figure 3. We may also assume that this same block has a second motion vector, this time equal to (-dx/2,-dy/2) which this time points to the k th frame (block C 3F ). Thus B3 can be reconstructed as the average of blocks C 3B and C 3F. This is rather similar to the way B frames are reconstructed, and its advantage is its relative simplicity, especially for hardware, since we may reuse the standard motion compensation architecture. A more accurate method would be to instead consider not the block at position (x,y) but at position (x+dx/2, y+dy/2) as is shown in Figure 4. Obviously here we are in a sense tracking block B 1 and are estimating where this block may be found inside the missing field frame. Such a technique is obviously more logical and accurate and in this case we may generate the block B 3 as the average of B 1 and B 2. Using this method it is evident that several pixels in the inserted frame are generated by more than one block, where as there might also be pixels that have no corresponding reference, thus leaving empty gaps inside the inserted field frame. For the former, the average of all corresponding pixels is selected, where as for the latter, blank pixels are predicted using the previous discussed motion compensated method. Such a method would be more accurate even though obviously more complicated. (x,y) B 1 (x+dx/2,y+dy/2) (x+dx,y+dy) B 3 field in k th frame B 2 Inserted field field in (k-1) th frame Figure 4: Tracking the motion of the interpolation block for the missing field We further propose combining the above two methods and to instead reconstruct the missing field frame not by selecting either of the regenerated fields, but their mean. Such a method can be seen as a special case of multihypothesis motion compensated techniques [14], which usually perform better than normal motion compensation. In this way the predicted missing fields will be more accurate and of higher quality. We name this method the Multihypothesis Motion Compensated Deinterlacing (MMCD). Such a technique, as would be also shown in the simulation section, benefits significantly from the smooth motion field that is generated using zonal based algorithms, but could also work with any other motion estimation algorithm. Zonal algorithms though have also the benefit that they require very little computation and it might be desirable in a practical deinterlacing system to perform additional motion estimation using smaller blocks, thus refining the motion field, and consequently the motion compensation technique discussed above. Thus in such a case zonal algorithms would be preferable. 3. MOTION CLASSIFICATION Even though using the motion compensation techniques described in the previous section could in most cases achieve relatively good results, we may further improve performance by also considering additional spatial and temporal information that may be available after the interleaving is performed. In addition, since unfortunately block based motion estimation algorithms cannot handle some specific types of motion, such as for example scaling and rotation, using such information we may further reduce some undesirable artifacts that may result from this process. Another problem of only using motion compensated techniques is that we do not in any way consider occlusion, which refers to the fact that there are no existing correspondence vectors for covered and uncovered background or newly appearing objects.

5 (a) (b) Figure 5: Motion classification (black non moving, white moving) in frame 4 of Foreman (a) before and (a) after filtering Thus, as an additional step, we propose classifying the types of motion for every pixel inside the missing fields, and depending on this classification to perform, or not, additional spatial and temporal filtering. Initially we classify pixels as stationary and non-stationary depending on the value of each motion vector (Figure 5). From our experiments we noted that enlarging the area in favor of the non-stationary region, using either an erosion operation or filtering, would further benefit the entire process (Figure 5b). If a pixel is characterized as stationary then no other operation is necessary and its value should be retained as is. Otherwise, we store the original value (MotCompY), and a second estimate (MedY) is taken using a spatial median filter, which only considers the current interleaved pixel and its adjacent pixels in the original preexisting field (Figure 6). Current Pixel Predicted interleaved field lines Original field lines Figure 6: Median filtering of the interleaved pixel. In a next step, a weighted temporal median filtering is performed on the interleaved lines based on the corresponding fields in frames k and (k-1). This operation basically results in a temporal filtering of these pixels. If for example the corresponding pixels inside the fields of the previous and future frame are PastY and FutureY respectively, this weighted temporal median filtering could be performed as: NewY = ( 3* MedY + MotCompY + Median( MotCompY, PastY, FutureY)) / 5, (1) In a final observation we have noticed that the performance of our estimation depends considerably on whether motion is happening in a horizontal direction, or is considered significant. In such a case we also perform an additional spatial averaging filtering with a filter H 1 (Equation 2), which can further reduce or eliminate possible artifacts due to faulty prediction. For all other pixels a second filter H 2 is used instead, that mainly benefits the value of the current interleaved pixel H = 1 = 1, H 2 10, (2)

6 If available, we may also consider the distortion values (SAD) from the motion estimation process to further assist in the classification of the pixels. For example if for a stationary block the distortion is larger than a threshold T this block should also be considered in the procedure described previously. The filters used could also be adaptively changed. Moreover, it is possible to consider edge-based line averaging algorithms, which can preserve more accurately line edges instead of the vertical filter we have used. Depending also on the complexity of the deinterlacing system we need to design, some of the above features could also be disabled. The entire process can be seen in Figure 7. Field k+1 1 Motion Estimation Motion Compensation Field k-1 1 Motion Classification Field Interleaving Field k 2 Spatial & Temporal Median Filtering Spatial Low pass Filtering Field k 1 Figure 7: Deinterlacing process as proposed in this paper 4. SIMULATION We have performed extensive simulations to determine the efficacy of the proposed deinterlacing method. For our experiments we have selected 3 CCIR interlaced sequences, Stefan, Bus, and Flower Garden. The PMVFAST algorithm was used as the motion estimation algorithm for these sequences with a search area of ±64. The original interlace and the deinterlaced frames for these sequences can be seen in Figures Obviously the visual quality for these frames is very good and there are few if any visual artifacts inside our reconstructed frames. Since though subjective quality and evaluation cannot always be considered reliable, we have performed an additional experiment with artificially interlaced sequences. We have selected five progressive CIF (30fps) sequences, namely sequences Foreman, Table Tennis, Stefan, Akiyo, and Miss America. These sequences were converted into interlace by dropping all even fields in the odd frames, and all odd fields in the even frames. Even and odd frames were then interleaved together to create a new interlaced frame. The sequences now were converted from 30frame per second sequences to a 30 field per second format. This procedure basically allows us to compare the reconstructed deinterlaced frame with the original reference frame, and thus more accurately be able to evaluate the performance of our algorithm. In our experiments, we have selected a block size of 16 8 for the motion estimation/compensation process. We may of course use a much smaller block, which could further increase performance since the prediction would be more accurate. We have also only used integer-pixel accuracy in the motion estimation, where as again, especially considering that motion compensation is performed based on an interpolated motion vector, half or even quarter pixel accuracy could benefit the estimation significantly. Furthermore, only corresponding fields were used in the motion estimation process.

7 We have performed two different experiments using the proposed technique, one using the Full Search (FS) algorithm and a second set of experiments using the PMVFAST algorithm. The technique was compared versus the line doubling (LDB) and line averaging (LAV) techniques. We have performed deinterlacing using three different methods. By using only the normal motion interpolation/compensation technique, the multihypothesis motion compensation (MMCD), and the multihypothesis motion compensation with the use of motion classification and filtering (MMCD+MCF). The PSNR results for all these cases can be seen in Table 2. Per frame results for these sequences can be seen in Figures No ME FS PMVFAST Sequence # of Frames LDB LAV None MMCD MMCD+MCF None MMCD MMCD+MCF Foreman Table Tennis Stefan Akiyo Miss America Table 2: Comparison of the proposed deinterlacing approach versus Line Doubling and Line Averaging It is obvious from this table that the LDB algorithm always yields the worst performance. Without the use of motion classification and filtering, it is evident that PMVFAST has a better performance than FS, as we would have expected. It is though noticeable that performance is not always as good as the LAV algorithm, as is apparent in the Foreman and Stefan sequences. Even worse, noticeable artifacts are also quite evident in the generated sequences. This can also be observed from the per frame results in Figures 8-12 where for specific frames PSNR is significantly lower than that of the LAV algorithm. We may though claim that MMCD performs significantly better than compared to the normal motion compensation technique. Instead, the use of motion classification and filtering appears to always have the best performance. Artifacts have been reduced significantly if not entirely eliminated, where as PSNR is significantly better than all other cases. For example, PSNR for Foreman, Table Tennis, and Akiyo sequences is almost 5dB, 7.5dB, and 7.5dB respectively higher than the LAV algorithm. Even for the Stefan sequence, which is a much more complicated sequence, PSNR is 1dB higher than LAV. Per frame results also suggest that the performance using this scheme is more reliable and robust than without the classification and filtering process. An additional observation is that PMVFAST does not always perform better than FS when classification is used. In particular, for sequences Table Tennis, Stefan, and Miss America, PSNR using FS is higher than that of PMVFAST. Our initial conclusion is that for these sequences more motion vectors are classified as nonstationary and thus filtering was more often used. This means that even in PMVFAST, if some of the currently classified as stationary blocks were instead classified as non-stationary, performance would improve further. At this point we should point out that the FS implementation we have used was not the zero-biased FS, most commonly used in standard systems such as for example in MPEG-4, but a straightforward implementation that considers all points equally. We would expect that in such a case, PSNR using the zero-biased FS would most likely degrade, since again more blocks would be classified as stationary, and less filtering would be performed. 5. CONCLUSION In this paper a new, highly efficient deinterlacing technique was presented. The method is based on a multihypothesis motion compensation approach and on motion classification/filtering. Performance in terms of PSNR and subjective quality suggests that the proposed technique is far more superior than other commonly used deinterlacing methods, such as for example the Line Doubling and Line Averaging techniques. Even though such an approach could work with motion vectors generated from any motion estimation algorithm, we further evaluate the performance of zonal based algorithms, and in particular PMVFAST, for deinterlacing. Results suggest that the regular motion fields generated by these algorithms can significantly help in the deinterlacing process, where as their insignificant complexity makes them a very good choice for a practical deinterlacing system. ACKNOWLEDGEMENT This work was funded by the RGC CERG HKUST6057/99E grant from the Hong Kong Government. REFERENCES 1. S. Pigeon and P. Guillotel, Advantages and drawbacks of interlaced and progressive scanning formats, HAMLET Rep., RACE 2110, 1996.

8 2. Microsoft Corp. Broadcast-enabled computer hardware requirements, in WinHEC 97, Broadcast Technologies White Paper, 1997, pp M. H. Lee, J. H. Kim, J. S. Lee, K. K. Ryu, and D. I. Song, A new algorithm for interlaced to progressive scan conversion based on directional correlations and its IC design, IEEE Transactions on Consumer Electronics, Vol. 40, No 2, pp , C. J. Kuo, C.Liao, and C. C. Lin, Adaptive interpolation technique for scanning rate conversion, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 6, No. 3, pp , Jun T. Chen, H. R. Wu, and Z. H. Yu, An Efficient Edge Line Average Interpolation Algorithm for Deinterlacing, Proceedings of SPIE, Visual Communications and Image Processing 2000 (VCIP 2000), Vol 4067, No 3 pp , Perth, Australia, Jun Y. Wang and S. K. Mitra, Motion/pattern adaptive interpolation of interlaced video sequences, Proceedings of EE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 91), pp , Toronto, Canada, May L. Vandendorpe, L. Cuvelier, B. Maison, P. Queluz, and P. Delogne, Motion compensated conversion from interlaced to progressive formats, Proceedings of EUSIPCO-94. Seventh European Signal Processing Conference, vol.3, pp , 1994, Lausanne, Switzerland 8. D. Hargreaves and J. Vaisey, Bayesian motion estimation and interpolation in interlaced video sequences, IEEE Transactions on Image Processing, Vol. 6, No 5, pp , A. J. Patti, M. I. Sezan, A. M. Tekalp, Robust methods for high-quality stills from interlaced video in the presence of dominant motion, IEEE Transactions on Circuits & Systems for Video Technology, vol.7, no.2, pp Apr A. M. Tourapis, O. C. Au, and M. L. Liou, Predictive Motion Vector Field Adaptive Search Technique (PMVFAST) - Enhancing Block Based Motion Estimation, to appear in proceedings of Visual Communications and Image Processing 2001 (VCIP-2001), San Jose, CA, Jan A. M. Tourapis, O. C. Au, and M. L.Liou, Fast Block-Matching Motion Estimation using Predictive Motion Vector Field Adaptive Search Technique (PMVFAST), ISO/IEC JTC1/SC29/WG11 MPEG2000/m5866, Noordwijkerhout, Netherlands, Mar A. M. Tourapis, O. C. Au, M. L. Liou, and G. Shen, Status Report of Core Experiment on Fast Block-Matching Motion Estimation using Advanced Diamond Zonal Search with Embedded Radar, ISO/IEC JTC1/SC29/WG11 MPEG99/m4980, Melbourne, Australia, Oct Optimization Model Version 1.0, ISO/IEC JTC1/SC29/WG11 MPEG2000/N3324, Noordwijkerhout, Netherlands, Mar M. Flierl, T. Wiegand, and B. Girod, Video codec incorporating block-based multihypothesis motion-compensated prediction, Proceedings of SPIE, Visual Communications and Image Processing 2000 (VCIP 2000), Vol 4067, No 3 pp , Perth, Australia, Jun 00 Figure 8: Per frame PSNR of deinterlaced Foreman using various algorithms

9 Figure 9: Per frame PSNR of deinterlaced Table Tennis using various algorithms Figure 10: Per frame PSNR of deinterlaced Stefan using various algorithms

10 Figure 11: Per frame PSNR of deinterlaced Akiyo using various algorithms Figure 12: Per frame PSNR of deinterlaced Miss America using various algorithms

11 (a) (b) Figure 13: Deinterlacing of the Stefan sequence (CCIR) using PMVFAST+MMCD+MCF (a) (b) Figure 14: Deinterlacing of the Flower Garden sequence (CCIR) using PMVFAST+MMCD+MCF (a) (b) Figure 15: Deinterlacing of the Bus sequence (CCIR) using PMVFAST+MMCD+MCF