Intermediate Image Interpolation using Polyphase Weighted Median Filters

Transcription

1 Intermediate Image Interpolation using Polyphase Weighted Median Filters Ortwin Franzen *a, Christian Tuschen b and Hartmut Schröder a a Circuits and Systems Group, University of Dortmund, Germany b Infineon Technologies Image & Video KG, Munich, Germany ABSTRACT A new algorithm for the interpolation of temporal intermediate images using polyphase weighted median filters is proposed in this paper. To achieve a good interpolation quality not only in still but also in moving areas of the image, vectorbased interpolation techniques have to be used. However, motion estimation on natural image scenes always suffers from errors in the estimated motion vector field. Therefore it is of great importance, that the interpolation algorithm possesses a sufficient robustness against vector errors. Depending on the input and output frame repetition rate different cyclically repeated interpolation phases can be distinguished. The new interpolation algorithm uses dedicated weighted median filters for each interpolation phase (polyphase weighted median filters) which are (due to their shift property) able to achieve a correct positioning of moving edges in the interpolated image, even if the estimated vector differs from the true motion vector up to a certain degree. A new design method for these dedicated error tolerant weighted median filters is presented in the paper. Other aspects like e.g. the preservation of fine image details can also be regarded in the design process. The results of the new algorithm are compared to other existing interpolation algorithms. Keywords: Image interpolation, weighted median filter, polyphase filter, filter design, integer programming. INTRODUCTION As there exist lots of different video standards with specific spatial as well as temporal sampling rates, format conversion between different standards is an important task in the field of video signal processing. Due to new image based services in a modern multimedia environment (e.g. displaying TV- or movie films together with a video telephony sequence and computer data on a single screen) and an increasing number of different display technologies (e.g. cathode ray tube, plasma displays, LCD) with specific spatio-temporal properties, the significance of image format conversion is even growing. Image format conversion can consist of a spatial conversion (scaling) and/or temporal conversion (picture rate conversion). A special case of image format conversion is de-interlacing for interlaced video sequences (proscan-conversion) which figures out as non-separable spatio-temporal interpolation problem. Since format conversion requires the interpolation of (spatial and/or temporal) intermediate samples, the quality of the interpolation result strongly depends on the utilized interpolation algorithm. In the following we will focus on temporal interpolation techniques. In order to achieve a high subjective interpolation quality in still as well as in moving areas of the image, motion vector based interpolation algorithms have to be used as this allows a correct positioning of moving objects in the intermediate images to be interpolated. These algorithms consist of two steps. In the first step, the motion between two successive input images is evaluated using an appropriate motion estimation algorithm (e.g. block matching). In the second step, the motion vectors can be used to position interpolation filters in the neighboring original images which allows an interpolation along the motion trajectory of the moving objects. Assuming pure linear motion and neglecting other effects like coverage and uncoverage etc., a perfect motion estimation and interpolation for every temporal position between the original pictures could be achieved. However, since these assumptions do not hold for natural images scenes, motion estimation always suffers from errors in the estimated motion vector field. Therefore it is of great significance that the interpolation algorithm possesses a certain robustness against erroneous motion vectors. The paper is organized as follows: In section a rough overview of vectorbased interpolation techniques for intermediate image interpolation is given. The motivation for the use of polyphase interpolation filters as well the new filter design technique for those filters are presented in section 3. A sample filter design and interpolation results for the new algorithm as well as other existing algorithms are discussed in section 4. Finally conclusions are drawn in section 5. * Correspondence: ortwin.franzen@uni-dortmund.de; Telephone: ; Fax:

2 . VECTORBASED INTERMEDIATE IMAGE INTERPOLATION In a temporal image format conversion, different phases depending on the input and output image repetition rate can be distinguished. In the part of Fig. the temporal positions of image sequences of different fields rates are depicted. It can be seen that for the conversion task from 50 Hz to 00 Hz, being very popular in Europe for flicker reduction, the output image is either at the same temporal position as the input image or it is temporally located exactly in the middle of two input images. But for a conversion from 50 Hz to 60 Hz six different interpolation phases can be distinguished which substantially differ in the temporal position of the output image according to the temporally neighboring input images. Figure. Temporal position of images for different image repetition rates () and definition of projection factors (right) The relative temporal position of the output imaging according to the neighboring input images can be characterized by so called projection factors p and p right with p denoting the relation of the distance from the output image to the previous input image to the distance T in between two input images. Comparably p right = p denotes the relative distance to the following input picture, as depicted in the right part of Fig.. Generally, in a conversion from the input image rate f in = T in to the output image rate f out = T out only k max different sets of projection factors occur with fout k = max gcd ( fin, fout ) () and gcd(a,b) denoting the greatest common divisor of a and b. Assuming, that the first image of the input and output sequence have the same temporal position, these projection factors are given by f in f in p = k k fout fout () p = p, right with k=0,,..., k max and x denoting the greatest integer number not being larger than x. Obviously the temporal image rate conversion can be divided into k max different cyclically repeating interpolation phases... Intermediate Image Interpolation Algorithms In order to interpolate a temporal intermediate image, different linear or nonlinear interpolation algorithms can be applied. A! pixel Φ ( x) =Φ( x, y) of the output image can e.g. be calculated as the weighted average of the corresponding pixels of the 0 surrounding input images F n and F n!!! Φ avg ( x) = ( p ) Fn ( x) + p Fn ( x). (3) Other interpolation algorithms make use of more sophisticated (spatio-) temporal linear or nonlinear filtering techniques. But in order to exploit the correlation between succeeding images not only in static but also in moving areas of the input image sequence, vectorbased interpolation techniques have to be used. The general concept of a vectorbased intermediate image interpolation algorithm is depicted in Fig.. Assume, an object moves translatory with a velocity v! real from the previous input image F n to the following input image F n. At the temporal position p Tin of the intermediate image, it has changed its position from the starting point in the previous image by a fraction p v! real of the total motion v! real between two input images. Accordingly the object moves by pright v! real between the temporal position of the intermediate image and the following input image. Therefore for maximal exploitation of the correlation between the input images, a vectorbased addressing of the input pixels of the interpolation filter is required (motion compensation).

3 Figure. General concept of vectorbased intermediate image interpolation To exploit not only temporal, but also spatial correlations in the input images the spatially surrounding pixels of the position addressed by the projected vector can be included into the interpolation filter. In addition, pixels of the output image which have already been calculated can be included into the interpolation filter (recursive filter elements). This leads to spatio-temporal interpolation masks as depicted in Fig.. In the following it is assumed, that an estimated motion vector v! est representing the motion from the previous input image to the following input image is available for every pixel of the previous image. Details of motion estimation algorithms for deriving those vectors are not discussed here, an example for such an algorithm can e.g. be found in 6. Simple examples for motion compensated interpolation algorithms are the motion compensated pixel shift 6 denoted by!!! Φ mcs ( x) = Fn ( x p vest ) (4) and the motion compensated weighted average which can be written as 6,0!!!!! Φ x = p F x p v + p F x+ p v. (5) ( ) ( ) ( ) ( ( ) ) mca n est n est In the case of wrong motion vectors the motion compensated pixel shift denoted by Eqn. 4 leads to a misspositioning of objects, which is recognized as strongly annoying artifact. A first improvement of the error tolerance is yielded by the motion compensated weighted average (Eqn. 5) leading in the case of faulty vectors to a blurring of the object which is less objectionable than the sharp misspositioning due to Eqn. 4. In 0, interpolation techniques for the vectorbased image interpolation based on a 3-Tap median filter are proposed. There, a so called static median is defined, denoted by!!!! Φ x = med F x ; F x ; Φ x (6) ( ) { ( ) ( ) ( )} sta n n mca with the Φ mca described in Eqn. 5 and med{x i } denoting the median (refer to 9,,3 ) of the samples x i. In addition, a so called dynamic median is defined in 0 by!!!!!! Φ x = med F x p v ; F x+ p v ; Φ x (7) { } ( ) ( ) ( ( ) ) ( ) dyn n est n est avg with the Φ avg described in Eqn. 3. The static median filter is robust against spurious vectors in stationary images areas since two of the three median inputs are not motion compensated. However, in areas with real motion it is assumed that the noncompensated input values strongly differ and therefore the third motion compensated input sample is chosen as filter output. But on the one hand this assumption is not always true, and some fallback to static interpolation may arise. And on the other hand, even if Φ mca is chosen as filter output, the already mentioned image blurring in the case of wrong motion vectors remains. The dynamic median filter yields a perfect interpolation for correct motion vectors since both vector addressed pixels have the same luminance value in this case and therefore are chosen as filter output. If on the contrary the motion vector is faulty, it is likely that the vector addressed pixels have so different luminance values, that the third (uncompensated) value is in the middle of the ranking and therefore chosen as filter output. This means, that the algorithm falls back to static averaging and therefore blurs the interpolated image the in the case of faulty vectors. Especially in high detailed image areas where small vector errors can lead to large luminance differences, these differences can cause the non motion compensated input value to be the output of the median filter at some positions and therefore serrated edges can occur 0. A more sophisticated algorithm based on median filtering of the output of Eqns. 6,7 and a linear combination of Eqns. 3 and 5 is also presented in 0... Intermediate Image Interpolation with Weighted Median Filters The interpolation algorithms presented before differ in their fallback behavior in the case of faulty motion vectors, but they have in common, that they don t have the ability to correct false motion vectors. In contrast, the interpolation algorithm pre-

4 sented in,,4 is able to correct faulty motion vectors up to a certain degree. The algorithm is based on the use of weighted median (WM) filters which can be defined by ywm ( xi ) = med { x w; x w;...; xn wn } (8) with x i denoting the input samples, w i denoting the filter weights and being the duplication operator, i.e. wx = xx,,..., x. (9) "#$#% w times For alternative definitions and a more detailed description of weighted median filters refer to 9,,3. The vector error tolerant algorithm presented in,,4 was designed for a conversion from 50 Hz interlaced sequences to 00 Hz interlaced sequences. The algorithms uses a vertical band separation according to a model for image perception which claims, that the human visual system can be modeled as a two channel system which one channel holding a low spatial but high temporal resolution (vertical lows) and a second channel with high spatial but low temporal resolution (vertical highs). The algorithm used for the temporal interpolation of the intermediate field Φ k in the lows channel can be explained using Fig., if F n and F n are assumed to be neighboring input fields of an interlaced input sequence (what implies that F n and F n have a different raster). The output of the interpolation filter is given by a weighted median including input pixels covered by the vector addressed weight masks W n in the previous as well as W n in the following input image. In addition, already calculated pixels of the actual output image covered by the weight mask W Φ can be included in the weighted median processing as recursive filter elements. Φ x! (circled in Fig. ) can be expressed by The calculation of the output pixel ( ) WM { }!!!!!! ( x) med W F ( x p v ); W F ( x ( p ) v ); WΦ ( x) Φ = + Φ wm n n est n n est wm with all pixels covered by one of the masks being feed into the filter with duplication according to the corresponding filter weight. If the vector addressed position in the previous or following field does not exactly match a discrete pixel position, linear spatial interpolation is applied. The loss of spatial resolution in the vertical direction due to linear interpolation can be gotten over because this happens to the vertical lows processing channel with implied low spatial resolution. In the vertical highs channel a 5-tap median filtering is applied. If the vector addressed vertical position in the previous or following field exactly matches a pixel position in this field, the corresponding pixel is feed twice into the median filter. If on the other hand the addressed position vertically lies between two pixels, each of this pixels is feed once into the filter. One additional (not motion compensated) pixel is taken from either the input field before the previous or after the following field depending on which of the fields has the same raster as the current output field. By this raster-dominant interpolation scheme a loss of vertical resolution which would be implied using a linear vertical raster interpolation is avoided as in the vertical highs channel this loss of resolution can lead to annoying flicker artifacts. The raster conversion required if the temporal position of the output field matches one of the input fields but both have a different raster is not discussed in this article since the main focus is addressed to temporal interpolation. For more details concerning the interpolation algorithm described above refer to,,4. It can be shown that the interpolation algorithm applying center weighted medians outperforms other existing algorithms for the case of a 50 Hz interlace to 00 Hz interlace conversion. This superior performance is especially caused by the motion vector error correction facilities of the proposed weighted median filters.... Correction of vector errors by weighted median filters The ability of correcting vector errors by using weighted median filters should be explained with the help of the example of a horizontally moving ideal vertical edge, for which a wrong motion vector was estimated. For the sake of a clear illustration this pure D interpolation problem is chosen as therefore pure D interpolation filters can be applied. As in the previous chapter, the interpolation of the intermediate image in a conversion from 50 Hz to 00 Hz which it located exactly in the middle between two input images (i.e. p = ) is regarded. Assume that an edge moves with a velocity of six pixel per frame (v real =6) whereas an erroneous velocity v est =0 was estimated by the motion estimation algorithm. Because the estimated motion is v est =0, the filter masks in the previous and following image are centered at the position of the pixel to be interpolated as depicted in the part of Fig. 3. If linear interpolation filters were used, a blurring of the edge in the intermediate image would be inevitable. But using a weighted median filter (here W n =W n ={,,,7,,,} is assumed), the moving edge is correctly interpolated at the position v real of half the horizontal motion although an incorrect velocity v est =0 was used to position the interpolation masks. This means, that in the example presented here an estimation error of x=6 with (0)

5 was corrected by the median filter. x = v v () real est Figure 3. Interpolation of a moving edge with standard () and polyphase (right) weighted median filters The interpolation algorithm presented in,,4 makes use of this shift property of (weighted) median filters. Apart from the behavior of weighted median filters according to correlated image signals (e.g. edges or areas) it is possible to analyze the behavior of those filters according to non-correlated images signals (e.g. irregular textures). It can be shown that by proper selection of the filter weights it can be achieved, that in the case of a correct motion estimation also non correlated fine image details can be kept in the interpolated image. By an analysis of the statistical properties of weighted median filters it was shown in that in the case of wrongly estimated motion vectors the weighted median interpolation algorithm tends to eliminate uncorrelated fine details. But Blume claims in that if it is not possible to correctly shift a fine details due to faulty vectors, an elimination of fine details in the intermediate image is to be preferred to a wrong positioning of this detail as the latter leads to the reception of blurring or even double contours whereas the further is only related with a less annoying loss of the perceived contrast of the fine detail. Another implementation of a vector based image interpolation using a weighted median filter can be found in 5. In a 3 non motion compensated but only motion adaptive interpolation approach is presented, which alters two weights of a FIRweighted-median-hybrid filter according to the amount of motion indicated by a motion detector. 3. POLYPHASE WEIGHTED MEDIAN INTERPOLATION FILTERS The interpolation algorithms presented in,,4 and 5 make use on the vector error correction ability of weighted median filters and have proved their superior performance for a conversion from 50 Hz to 00 Hz where the temporally interpolated output images lie exactly in the middle of two input images, i.e. p =. The filter masks proposed in,,4 are chosen equal for both the previous and following picture. In 5 a choice between two alternative filter masks is specified for each of the previous and following picture, but the choice only depends on whether the vector addressed position vertically exactly matches an existing input field line or lies between two lines. In both algorithms the sum of weights in the filter masks for the previous and following image are equal. However, if other conversions are to be applied like e.g. a conversion from 50 Hz to 60 Hz, different cyclically repeated interpolation phases with projection factors according to Eqn. appear. In most of these phases the interpolated image does not lie exactly in the middle of the two neighboring input images but closer to one of them (refer to Fig. ). Therefore the influence of vector errors on the positioning of the filter masks in each of the surrounding images is not symmetric any more. Suppose e.g. the interpolation of an intermediate image at a temporal position corresponding to a projection factor p = 5 (and therefore p right = ) has to be carried out and a motion vector error of up to a magnitude of six can occur ( x 6 max =6). As the positioning error of the filter mask is limited to p xmax in the previous image and to pright xmax in the following image, a positioning error of maximum two can occur in the previous image in contrast to a positioning error up to ten in the following image. Due to the unequal influence of vector errors on each of the surrounding input images an interpolation algorithm which uses different filter masks for each interpolation phase and for each of both surrounding input images seems to be self-evident. Since such weighted median filters imply one special filter mask for each interpolation phase, they are called polyphase weighted median filters within this paper. Another motivation for using polyphase weighted median filters instead of one common filter mask for all interpolation phases is given by the following example. Again regard the conversion from 50 Hz to 60 Hz, the interpolation phase with 5 p = (and therefore p 6 right = ) and an ideal edge moving with a velocity v 6 real =6 pixel/image. As in the previous examples an erroneous estimation velocity v est =0 is assumed which leads to an identical positioning of the interpolation masks in the previous as well as following input image. An interpolation using a weighted median with the same interpolation masks W={,,,7,,,} in the image F n and F n according to,,4 yields the interpolation result shown in the part of Fig. 3. Obviously the interpolation leads to a positioning of the moving edge on half of the real motion in the output image which 6

6 would be correct if p =, but leads to a faulty positioning for the interpolation phase p = and therefore to a wrong 6 motion portrayal according to this interpolation phase. In contrast, using the filter masks depicted in the right part of Fig. 3, being especially designed for the regarded interpolation phase, a correct positioning of the moving edge according to this interpolation phase and therefore a correct motion portrayal of this moving edge can be achieved. The examples presented show, that the use of polyphase weighted median filter masks is mandatory to achieve a correct positioning and therefore a correct motion portrayal of moving edges (which can serve as a model for large moving objects) even for faulty motion vectors in the case of a format conversion with more than two ( p = 0 and p = ) interpolation phases. 3.. A new Design Technique for Vector Error Tolerant Weighted Median Filters As there existed no suitable design methods for error tolerant weighted median interpolation filters, heuristically designed filters have been used in interpolation algorithms presented in the literature (,,4 and 5 ). Approaches presented in 3 based on minimizing the mean squared error between an interpolated image sequence and a reference image sequence using evolution strategies as optimization techniques suffer from a very high computational load. In addition, the filters derived from those design techniques are adapted to the special properties of the test sequence and therefore do only allow to transfer the design results to image- or motion-situations not differing from the ones in the test sequence. A second approach presented in 8 allows an easy design of error tolerant interpolation filters in the field of intermediate views interpolation for stereoscopic sequences, but offers only very restricted design possibilities. Since it is limited to center weighted medians (i.e. all weights except of the central weight equal one), an estimated disparity of zero is presupposed and rounding effects for fractional values of p vreal or pright vreal are neglected, the space of possible solutions is severely restricted, so that the design method fails in many cases in designing appropriate error tolerant interpolation filters. Due to the disadvantages and limitations of existing filter design techniques, a new design technique is presented in this paper, consisting of two design steps. In the first step, the conditions for the filter weights are derived, which have to be fulfilled, if the filter should be able to correct faulty motion vectors. In the second step, an optimum interpolation filter is designed using integer programming Deriving the conditions for the filter weights In order to design a vector tolerant weighted median interpolation filter, the constraints on the filter coefficients to achieve the demanded error tolerance have to be formulated. The notation used in this paper is explained using Fig. 4. Again a moving edge with v real =6 and the interpolation phase with p = is considered, whereas in contrast to recent examples an estimated velocity of v est =3 is assumed. Let x c denote the correct position of the moving edge in the output image Φ k and x n resp. x n denote the position of the moving edge (according to the real velocity v real ) in the input images F n resp. F n. The positions f n and f n are the central positions of the filter masks in the previous and following input image if the output pixel position x c is to be interpolated. Figure 4. Notation for the filter design for position x c to be interpolated Then a Difference of the Central Position of the filter mask CP for the previous and following image can be defined as CP = x f n n n CP = f x = x CP n n n n. For a spatially continuous image representation, Eqn. can be written as ()

7 ( ) ( ) ( ) ( ) CP = x + p v x + p v = p v v = p x n c right real c right est right real est right CP = x p v x p v = p v v = p x = x CP n c est c real real est n, and therefore CP n and CP n can be interpreted as the estimation error projected on the previous and following images. But as the new filter design technique is based on a spatially pixel discrete schema, a mapping of fractional values of f n, f n, x n and x n to integer positions is presupposed in the following considerations. This can e.g. be realized by CPn = real int ( pright vreal ) real int ( pright vest ) (4) CPn = x CPn assuming without loss of generality v c being the origin of the x-axis (i.e. v c =0) and with realint() being an arbitrary mapping function from real to integer values (e.g. rounding towards 0 or towards ). The definition of CP n according to the second line of Eqn. 4 guarantees that the sum of CP n and CP n equals the total estimation error. The notation for the weights of the filter masks in the previous and following image is depicted in Fig. 5 with L n and L n denoting the filter expansion measured from the central position (resulting in a total filter mask length of L n + and L n +, respectively). Recursive filter elements are not regarded in the filter design process. (3) Figure 5. Notation for the filter masks Let for the filter design process the moving edge (regarded as model for large moving objects) being a binary transition from a high luminance (F(x)=, denoted by H ) to a low luminance (F(x)=0, denoted by L ). Due to the stacking property 9 of weighted median filters this model represents all other ideal transitions between arbitrary luminances. As x c denotes the correct edge position (refer to Fig. 4) in the intermediate image according to the actual interpolation phase, for horizontal positions in the output image being smaller than x c a high luminance (H) has to be interpolated whereas for positions greater or equal than x c the luminance to be interpolated is low (L) as depicted in Fig. 6. Figure 6. Condition for correct intermediate image interpolation To simplify the following considerations the notation SW ( M, X x ) with M {W n ;W n } and X {L;H} is introduced, where e.g. SW ( W, n L xc ) denotes the sum of the filter weights in the filter mask W n which cover a pixel of the input image F n with low luminance if the position x c of the output image is to be interpolated. Then the conditions for a correct interpolation can be formulated as ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) x x : SW W, H x + SW W, H x < SW W, L x + SW W, L x c n n n n x < x : SW W, H x + SW W, H x > SW W, L x + SW W, L x. c n n n n Because it can be shown, that the inequalities for x < x c are weaker than the inequality for x = x c and the inequalities for x > xc are also weaker than the one for x = xc, Eqn. 5 can be simplified to ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) x = xc : SW Wn, H xc + SW Wn, H xc < SW Wn, L xc + SW Wn, L xk (6) x = xc : SW Wn, H xc + SW Wn, H xc > SW Wn, L xc + SW Wn, L xc. In general, the input pixels of the filter mask W n are H for all pixel positions x < xn and L for all pixel positions x xn and obverse the inputs of the filter mask W n are H for x < xn and L for x xn since x n and x n denote the edge positions in the input images. If the position x c is to be interpolated, Eqn. holds and therefore xn = fn CPn resp. xn = CPn + fn. (5)

8 Additionally regarding the indexing of the filter mask (i.e. that the pixel at the position f n +k corresponds to the filter weight w n (k) and identically the pixel at f n +k corresponds to w n (k), refer to Figs. 4 and 5), the first line of Eqn. 6 can be written as x = x : w i + w i < w i + w i () () () (). (7) c n n n n i< CPn i< CPn i CPn i CPn If the position x c is to be interpolated, the filter masks are shifted one pixel to the whereas the edge position x n in the previous as well as x n in the following image stay the same. Therefore the index of the filter weights corresponding to a pixel with luminance H resp. L increases by one, which allows to rewrite the second line of Eqn. 6 to x = x : w i + w i > w i + w i () () () (). (8) c n n n n i< CPn + i< CPn+ i CPn + i CPn Designing an error tolerant filter using integer programming It comes out that the conditions for a correct interpolation of the moving edge can be formulated by two inequalities for the filter weights only depending on the projected estimation errors CP n resp. CP n. As these are not the typical input variables for a filter design technique, a mapping of characteristic variables to those projected estimation errors is necessary. The filter design task addressed in this paper can be formulated as: Design an error tolerant filter which (for the estimated velocity v est ) is able to fully correct a motion estimation error of x max for the interpolation phase with a given p right. Fully correction of the motion estimation error x max for an estimated velocity v est means, that for all real velocities in the range of [v est x max ; v est + x max ] the edge is interpolated at the correct position. This leads with x [ x max ; x max ] and x being integer to a set of x max + pairs of inequalities according to Eqns. 7 and 8, each containing values for CP n and CP n with respect to CPn = real int ( pright vreal ) real int ( pright vest ) = real int ( pright ( vest + x) ) real int ( pright vest ) (9) CPn = x CPn. All weighted median filters whose weights fulfill the whole set of 4 x max + inequalities are solutions of the filter design task formulated above. In order to achieve a hardware expense being as low as possible, a minimization of the sum of the filter weights seems appropriate. This leads to the formulation of the filter design problem as an integer program Ln n () n () cost( ) n( ) minimize z = cost i w i + j w j i= Ln j= Ln L L n n ki, n ( n ) + k, j+ L ( ), n + n n (0) i= 0 j= 0 n () 0, n( ) 0 (), ( ) n n L under theconstraints a w i L a w j L k M w i w j w i w j integer with M denoting the number of inequalities and cost(j) denoting a cost-function (which can e.g. be chosen to increase monotonically from i =0 and j =0). The values a k,j {,} result from a reformulation of Eqns. 7 and 8 according to the shape defined in Eqn. 0. An appropriate choice for the L n resp. L n was found to be given by the maximum of all CP n resp. CP n occurring in the generation of the set of inequalities (Eqns. 7 and 8) for all x [ x max ; x max ] Design of D-filters So far, only the design of one dimensional filter masks has been addressed in this paper. However, in general an intermediate image interpolation requires two dimensional filter masks in order to correct motion estimation errors in the horizontal as well as vertical direction. In this paper, the design of D-filters is limited to cross-shaped filters and the corresponding filter weights are depicted in Fig. 7. The filter length and therefore the maximal correction interval in the horizontal and vertical direction do not need to be identical. These filters are designed to be able to correct a horizontal estimation error for a horizontally moving ideal vertical edge (with infinite extension in the vertical direction) as well as a vertical estimation error for a vertically moving ideal horizontal edge (with infinite extension in the horizontal direction). E.g. for the correction of a horizontally moving edge, all vertical filter coefficients w n (0,i) cover the same input pixel value due to the stated infinite extension of the horizontally moving edge in the vertical direction. Corresponding contemplations can be done for the filter mask W n for the horizontally moving edge and for both filter masks for a vertically moving edge.

9 As therefore both estimation errors can be considered separately, the inequalities for the D design can be derived as the conjunction of the inequalities resulting from the constraints for each of the horizontal and the vertical estimation error correction. Figure 7. Cross shaped D-filter masks Inclusion of additional constraints By optimally solving the set of inequalities presented in the previous sections using an integer program, an error tolerant weighted median filter with D or D filter masks can be designed. In order to achieve a good interpolation quality, not only the behavior of the filter according to moving edges, but also to other criterions as fine image details is to be considered. Those additional demands on the filter behavior can easily be included in the filter design, if they can be expressed as a linear inequality of the filter weights. Two of the many additional demands included in the practical filter design technique should be listed exemplary in the following. Preservation of fine image details in case of correct motion estimation: In order to achieve a high subjective quality of the interpolated images, not only the correct positioning of edges (modeling large objects), but also the preservation of fine details is of great significance. By introducing an additional constraint for the filter coefficients according to L n n ( 0,0) + ( 0,0 ) > (,0) + (,0) + ( 0, ) + ( 0, ), () w w w i w i w j w j n n n n n n i= j= it can be guaranteed that in the case of a correct motion vector fine image details are kept, since then both pixels covered by the center weights have the same luminance and as their cumulative weights exceed the sum of the other weights, this luminance becomes the output of the weighted median filter. Higher total weight for one filter mask: If p < holds, the output image is closer to the input image F n than to F n. Therefore it seems appropriate, that the input pixels of the image F n have the majority in the weighted median filter. This can easily be achieved by an additional constraint w ( i, j n ) w n( i, j ) c > +, () with c being a nonnegative constant corresponding to the desired level of overweighting for the mask W n. Comparably a constraint demanding an overweight for W n can be introduced, if p >. 4. SAMPLE DESIGN AND SIMULATION RESULTS A sample design of a polyphase weighted median (WM) filter for a conversion from 4 Hz (progressive) to 60 Hz (progressive) using the new design technique presented in this paper is depicted in the part of Fig. 8. The filter is designed to be able to fully correct a motion error up to x max =3 and y max =3, whereby v est =0 was assumed and the detail preservation criterion according to Eqn. as well as the overweighting according to Eqn. (with c=4 for p =0. or 0.8 respectively c= for p =0.4 or 0.6) were considered in the design process. In the right part of Fig. 8 the weighted median masks according to,,4 are shown which are not dependent of the actual phase. Note, that in contrast to the polyphase weighted median filter, a recursive filter element is used in the mask proposal of,,4. In the following Figs. 9-0 some interpolation results for different interpolation algorithms described in this paper are presented. Beneath the neighboring input images F n and F n of the image Φ to be interpolated, the interpolation results of the weighted average filter (Φ avg ), the motion compensated average filter (Φ mca ), the static and dynamic median filter (Φ sta and Φ dyn ), the weighted median filter (Φ WM ) according to,,4 and three polyphase weighted median filters (Φ poly-wm ) with different maximum correction intervals are depicted in the figures. All polyphase weighted median filters were designed regarding L

10 Eqn. and (with c=4 for p =0. or 0.8 respectively c= for p =0.4 or 0.6). A vertical or two dimensional band separation was not used in any of the algorithms. The motion vectors for each simulation were derived using a parallel predictive 4 motion estimation algorithm and have pixel accuracy. Figure 8. Sample design for polyphase WM and standard WM It can be seen in the top line of Fig. 9 that the first four algorithms cause a blurring in the horizontally moving huge black object at the right and in the parts of the horizontally moving black vertical edge around the stationary letter U due to wrongly estimated motion vectors. By using weighted median filters, this blurring can avoided. The polyphase weighted medians with a horizontal correction interval of minimum four can even fully correct the wrong vector and achieve a correct positioning of both black objects. The best interpolation result can be achieved with the last filter, whose horizontal correction interval was chosen to be larger than the vertical one because the motion in this image sequence is mostly horizontal. This filter can avoid the small artifact above the U which arises for the polyphase WM with x max = y max =4 because the image modeling by a pure horizontal or vertical edge used in the filter design procedure is violated for the filter with x max = y max =4 at the artifact position. In the second line of Fig. 9 it can be seen, that only the weighted (polyphase or non-polyphase) median filters can avoid double contours at the women s chin. The dynamic median filter and all WM filters erase the top of the dot of the letter i, but this is less annoying than the artifacts above the dot introduced by the motion compensated weighted average filter. F n F n Φ avg Φ mca Φ sta Φ dyn Φ WM Φ poly-wm x max =, y max = Φ poly-wm x max =4, y max =4 Φ poly-wm x max =4, y max = p=0.6 p=0.4 p=0.4 Figure 9. Exemplary interpolation results for mat_phone (with stationary letters stamped in) and football

11 The third line of Fig. 9 again makes clear, that the fallback mode of all interpolation filters but the (polyphase-) WM filters can lead to a blurring of objects (here: the head of the football player). The non-polyphase WM filter does not introduce blurring, but deletes the part of the head and therefore the head becomes to thin whereas the polyphase WM with x max = leads to a head being to thick, since apparently the motion estimation error exceeds the correction abilities of this filter. Only the polyphase WM filters with x max =4 allow a correct positioning of the head. However, the fine texture in the grass is kept slightly better by the motion compensated average or the static/dynamic median than by the WM filters since the latter include a spatial filtering of the luminance values from the input images due to the spatial extension of the filter masks. Some interpolation results for the sequence mat_house are presented in Fig. 0. Since this sequence is dominated by vertical motion, the polyphase WM filter with asymmetrical correction interval in the last column of Fig. 0 is designed to have a higher vertical motion error correction ability. It can be seen in the interpolation examples for a vertically moving letter A and a vertically moving light bulb, that all filters but the weighted median filters include a strongly annoying blurring in the areas with wrongly estimated motion vectors. In those critical areas, the non-polypase WM tends comparably to the football sequence to a deletion of parts of the moving object (which is very annoying at the light bulb), whereas the polyphase median filter with x max = leads to a misspositioning of those object parts since in this example its maximum correction interval is lower than the estimation error. By using polyphase WM filters with a suitable correction interval, a correct positioning of the moving object and a reduction of the interpolation artifacts can be achieved. F n F n Φ avg Φ mca Φ sta Φ dyn Φ WM Φ poly-wm x max =, y max = Φ poly-wm x max =4, y max =4 Φ poly-wm x max =, y max =4 p=0.6 p=0.8 P=0.8 Figure 0. Exemplary interpolation results for mat_house Polyphase weighted median filters were also tested in the field of a temporal conversion of interlaced input images (e.g. for the conversion from 50 Hz interlace to 60 Hz interlace/progressive). An improvement of the interpolation quality was achieved by a replacing the weighted median filters in the lows channel of the algorithm proposed in,,4 by polyphase weighted median filters while keeping the interpolation method in the highs channel unchanged. 5. CONCLUSION In order to achieve a good interpolation quality for a vectorbased temporal intermediate image interpolation, it is of great significance that the interpolation algorithms possesses a sufficient robustness against errors in the motion vector field, since motion estimation on natural image scenes always suffers from such errors. Many interpolation algorithms are proposed in the literature, but most of them do in case of faulty motion vectors only implement a fallback mode which aspires to suppress or smooth interpolation artifacts. In contrast, the algorithm presented in is able to correct motion estimation errors for moving edges (which serve as model for larger image objects) up to a certain degree by using a weighted median as interpolation filter. However, the presented interpolation scheme does only provide the desired correction ability, if the temporal position of the output image to be interpolated lies exactly in the middle between two input images (like in a conversion 50 Hz to 00 Hz). For many conversion tasks (e.g. 4Hz to 60 Hz or 50 Hz to 60 Hz), several cyclically repeated interpolation phases can be distinguished according to the relative temporal position of the output image to the surrounding input images, and in maximal one of this phases the filters proposed in achieve the desired correc-

12 tion ability. Therefore an approach using a set of weighted median filters each being error tolerant for exactly one of the interpolation phases is proposed in this paper (polyphase weighted median filters). A new design technique for error tolerant weighted median filters based on optimally solving a set of linear inequalities for the filter weights by integer programming is presented in this paper. Additional design constraints, like e.g. the preservation of fine image details for correct motion vectors can easily be included in the design process. The interpolation results of polyphase weighted median filters with different maximal correction interval designed by the new technique are compared to the weighted median approach of and other interpolations algorithms presented in the literature. In contrast to the other algorithms, the polyphase weighted median filters are able to correct motion vector errors for edges and large objects up to a certain degree and therefore in general produce less interpolation artifacts. The maximal correction ability of those weighted median filters increases with the size of the filter masks. But on the other hand larger filter masks can lead to a stronger suppression of fine image details in case of faulty motion vectors. In addition larger filter masks hold an increased danger of interpolation errors due to a violation of the model of a moving edge being the basis for the filter design (e.g. if the filter mask covers more than one image object). Therefore an appropriate choice of the correction interval is necessary. The best interpolation result can be achieved, if the correction intervals in x- and y-direction are adapted with respect to the amount of motion in the corresponding direction. Future research will be addressed to an adaptation of the filter characteristics (maximal correction interval for x- and y-direction, detail preservation etc.) to the (local or global) image and vector field properties, e.g. according to the local homogeneity, the amount of motion or other values provided by the motion estimation algorithm (like the displaced frame difference etc.). In addition, an inclusion of a (linear or nonlinear) band separation whereby polyphase weighted median filters with different maximal correction intervals are applied for each of the bands will also be examined in the future. REFERENCES. H. Blume, Nonlinear Vector Error Tolerant Interpolation of Intermediate Video Images by Weighted Medians, Signal Processing: Image Communication, 4, pp , 999. H. Blume, Vectorbased Nonlinear Upconversion Applying Center Weighted Medians, IS&T/SPIE Symposium on Electronic Imaging, Sub Conference Nonlinear Image Processing VII, San Jose, USA, January 9 th -February nd, pp. 4-53, H. Blume, O. Franzen, M. Schmidt, Optimizing Video Signal Processing Algorithms by Evolution Strategies, Proc. of the 5 th FUZZY Days, Dortmund, Germany, April 8 th -30 th, pp , H. Blume, H. Schröder, Image Format Conversion Algorithms, Architectures, Applications, Proc. of the IEEE Workshop on Circuits and Systems for Signal Processing, Mierlo, The Netherlands, November 7 th 9 th, pp. 9-37, M. Braun, M. Hahn, J.-R. Ohm, M. Talmi, Motion-Compensating Real-Time Format Converter for Video on Multimedia Displays, Proc. of the IEEE 4th International Conference on Image Processing (ICIP-97), Santa Barbara, CA, USA, October 6 th -9 th, pp. 5-8, G. de Haan, IC for Motion-Compensated De-Interlacing, Noise Reduction, and Picture-Rate-Conversion, IEEE Transactions on Consumer Electronics, 45, pp , G. de Haan, P. Biezen, O.A. Ojo, An Evolutionary Architekture for Motion-Compensated 00 Hz Television, IEEE Transactions on Circuits and Systems for Video Technology, 5, pp. 07-7, M. Lück, H. Schröder, Error Tolerant Interpolation of Intermediate Views for Real-Time Applications, IS&T/SPIE Symposium on Electronic Imaging, Sub Conference Stereoscopic Displays and Virtual Reality Systems VI, San Jose, USA, January 3 rd -9 th, pp. 08-8, I. Pitas, A.N. Venetsanopoulos, Nonlinear Digital Filters, Kluwer Academic Publishers, Boston, O.A. Ojo, G. de Haan, Robust Motion-Compensated Video Upconversion, IEEE Transactions on Consumer Electronics, 43, pp , 997. A. Schrijver, Theory of Linear and Integer Programming, John Wiley & Sons, Chichester, 000. R. Yang, L. Yin, M. Gabbouij, J. Astola, Y. Neuvo, Optimal Weighted Median Filtering under Structural Constraints, IEEE Transactions on Signal Processing, 43, pp , L. Yin, R. Yang, M. Gabbouj, Y. Neuvo: Weighted Median Filters: A Tutorial, IEEE Transactions on Circuits and Systems, 43, pp. 57-9, 996