Geometric transform motion compensation for low bit. rate video coding. Sergio M. M. de Faria

Transcription

1 Geometric transform motion compensation for low bit rate video coding Sergio M. M. de Faria Instituto de Telecomunicac~oes / Instituto Politecnico de Leiria Pinhal de Marrocos, Polo II-FCTUC 3000 Coimbra, Portugal sergio@it.uc.pt Abstract: In this paper it is shown that the combination of spatial transformation and image segmentation can be used to compensate non-uniform intensity changes in moving scenes. This method eciently tracks movements such that, for low bit rate video coding, the motion vectors alone can be used to represent a complex motion of moving objects. The compression eciency of the presented method is far superior to that of the conventional block matching motion estimation, and the computational complexity is still aordable due to the use of fast transformation and interpolation algorithms. Keywords: Video coding, image compression, low bit rate, motion compensation, image warping, geometric transforms. 1. Introduction Interframe predictive coding is one of the most powerful image coding technique which can eliminate redundancy in moving scenes [1, 2]. In a general interframe predictive coder, the input frame is subtracted from the predicted preceding frame and the dierence signal is coded and transmitted. As information rate is associated with the dierence signal, its reduction will reduce the bit rate and such requirement necessitates a more ecient predictor. An ecient predictor can be designed using pixels in the previous frame that have been displaced by the motion of moving objects from frame to frame. Presently, the most popular motion estimation technique is the Block-Matching Algorithm (BMA) [3] that has been employed in the standard video codecs such as H.261 and MPEG. The BMA method relied on the assumptions that: Escola Superior de Tecnologia e Gest~ao - Morro do Lena Leiria, PORTUGAL i) motion of moving objects is purely translational, ii) illumination is uniform in the spatial and temporal domain, and iii) masking between objects and uncovered background can be neglected. However, these assumptions are unrealistic. In practice, motion has a complex nature and can be decomposed into translation, rotation, shear, expansion and other deformation components. Also, illumination changes are mainly non-uniform and the uncovered background in image coding applications cannot be easily ignored. To address the incompatibility between these unrealistic assumptions and practical motions, methods that employ spatial transformations have been proposed [4, 5, 6]. The Block Matching with a Geometric Transformation (BMGT) is capable of complex motion compensation with commendable bitrate improvement. This method is so ecient that most of the time only the motion vectors are required to be sent without the need for the error dierence signal. In this paper, to show the effectiveness of the BMGT technique, the image sequence is coded with only motion vectors. To improve compression further, motion compensation is applied on variable block sizes using image segmentation. In contrast to the continuous mesh concept used in [4, 6], in the presented method, as well as in [5], the optimal mappings are not constrained by the neighbouring mapped blocks. This feature makes it possible that a quadrilateral of any size and shape from the previous frame to be mapped into the current block, hence compensating for a more general complex motion. This part will be discussed in the experimental results given in section 4.

2 The paper is organised as follows. Section 2 gives an overview into the BMGT implementation and its complexity requirement; also various techniques that reduce the complexity level of the motion estimator are described. In section 3, quadtree segmentation of picture frames into variable size blocks to enhance the bit rate performance is presented. Experimental results and conclusions are given in sections 4 and 5, respectively. 2. Block matching with geometric transformation In a typical BMA, picture frames are partitioned into regular rectangular blocks of pixels and each block in the current frame is searched for a best matched block in the previous frame. The criterion for the best-match is the minimisation of the matching distortion function using either the Mean Absolute Error (MAE) or the Mean Square Error (MSE) functions [7]. The BMA adopts a motion model that describes object movements in the form of pure translation which is inadequate to represent true motion. In the block matching with geometric transform, BMGT, blocks of pixels in the current frame are deformed appropriately to best model the complex motion. The deformation process is achieved through the use of geometric mappings which provide a more general approach to motion estimation and enable complex motion to be tracked eciently and ultimately enhance the bit-rate performance. The parameters of the spatial trans- P0(u,v) P3(u,v) P1(u,v) P2(u,v) Forward mapping Inverse mapping P0(x,y) P3(x,y) Figure 1. Geometric mappings. P1(x,y) P2(x,y) formation are obtained by solving a set of linear equations relating the four vertex coordinates (i.e. P0, P1, P2 and P3) of a block in the current frame to the mapped quadrilateral's vertex coordinates (i.e. P0, P1, P2 and P3) in the previous frame as shown in Fig. 1. Similar to BMA, the bestmatched quadrilateral is found by displacing each vertex of the previous block, such that the distortion measure inside a search window is minimised. This is analogous to deforming the corresponding block in the previous frame to obtain a shape that best matches the current block Geometric Transformation Geometric transforms that possess the required mapping functions for complex motion are: af- ne, bilinear and the perspective transforms. The ane and the perspective transforms are planar mapping functions while the bilinear transform is non-planar [8]. For motion compensation applications the ane transform is not preferred as its degree of freedom is limited and is not capable of handling true three-dimensional motion. Also, the perspective transform is less attractive as it is more computationally intensive than the bilinear counterpart and still their peak-to-peak signal to motion compensated noise ratios (PSNR) are comparable [5]. Hence, special emphasis is placed on the bilinear transform due to its simplicity and its ability to facilitate quadrilateral-toquadrilateral mappings. Its non-planar mapping characteristic makes it a better model for head and shoulders features Bilinear transform The bilinear mapping function that relates pixels of the current frame to the previous ones is dened as X(u; v) = a0 + a1u + a2v + a3uv (1) Y (u; v) = b0 + b1u + b2v + b3uv (2) where (x,y) and (u, v) are the pixel coordinates of the mapped quadrilateral in the previous and the current frames, respectively. It is capable of handling four-corner mapping problems. However, it only preserves lines that are either horizontal or vertical in the source image as straight lines and points along these orientations remain equispaced in the mapped image [8]. Lines other than the two orientations mentioned are not mapped as lines but rather as quadratic curves. To determine the mapping parameters, a set of linear equations relating the coordinates of a square block in the current and the best-match quadrilateral in the previous frame are solved. The computational load is relieved by osetting the coordinates of every current square block to the origin, as shown in Fig. 1. Referring to this gure, the

3 mapping parameters are given by a0 = x0 a1 = x 1? x0 15 a2 = x 3? x0 15 (3) a3 = x 2? x3? x1 + x0 225 In the conventional BMA, the components of the motion vectors are transmitted alongside the frame dierence signal. However, in the BMGT method transmission of motion information can take dierent forms. Either the mapping parameters, a i, themselves are transmitted or the displacement vectors x = x i? x j ; y = y i? y j for each vertex of the best-matched quadrilateral. The transmission of the mapping parameters, a i, enables the decoder to reconstruct the quadrilateral without recurring to parameter calculations. However, if the displacement vectors were sent, recomputation of the parameters at the decoder is required. While transmission of mapping parameters requires large bits as they are continuous valued, the displacement vectors are discrete valued which incur much lower overhead than the former. In this paper the later method is employed. Mapping of a square block of pixels to a quadrilateral usually results in image resampling [8]. The resampled grid does not generally coincide with the input sampling grid which is discrete as the range of the continuous mapping function is a set of real numbers. The simplest way to do it is to approximate the pixel value by the nearest pixel value, called Nearest Neighbour Interpolation. But this method lacks accuracy, because it does not take into consideration the values of other pixels in the neighbourhood. However with the Bilinear Interpolation one can approximate the intensity of a pixel, P, using the intensity of its four nearest neighbours P a -P d, as described by Eqn. 4 P = (P b? P a )p x + (P d? P a )p y + (P c? P d? P b + P a )p x p y + P a (4) where P is the interpolation sample, P a, P b, P c and P d are the four nearest neighbours Arithmetic operations for each mapping The matching criterion used to quantify the picture quality is Mean Absolute Error (MAE). For each matching, the conventional BMA requires this error function to be evaluated only once, which needs roughly 2B 2? 1 simple additions/subtractions and two multiplications. On the other hand, BMGT in addition to this matching function requires the spatial mapping, which is more complex. The additional computations include the previously described bilinear interpolation (for every possible geometric mapping), computation of the motion parameters (only once per quadrilateral mapping) and the transform mapping. The bilinear mapping involves mapping a quadrilateral of pixels in the previous frame onto a block of pixels in the current frame. This stage comprises numerous computations, since every pixel has to be transformed from one quadrilateral to another, using Eqns. 1 and 2. This process can be greatly simplied with the scanline technique, where pixels belonging to the same row or column of a block are transformed one after another. By simple manipulation of these equations, the transformed coordinates of a pixel can be calculated from the transformed coordinates of the preceding adjacent pixel Mappings for each compensated block For a maximum object movement of! pixel/ frame, the maximum number of search points for each search vertex of a quadrilateral, p, is given by p = (2! + 1) 2 : (5) Since the four vertices of the quadrilateral may be displaced simultaneously, the number of possible mappings, M, is given by M = p 4 = (2! + 1) 8 : (6) For!=16 pixel/frame, this full search BMGT may perform up to 1: quadrilateral mappings. In addition to the mapping operations, each MAE test executed for a new mapped quadrilateral takes 2B 2? 1 simple additions/subtractions and two multiplications. Even considering that many quadrilateral shapes are never reached due to its geometric degeneration or due to the criteria for optimal mapping, a real-time implementation using BMGT seems to be impractical. Hence, it is crucial to reduce the number of quadrilateral mappings to an aordable computational cost. In the following section it is shown that a combination of two-stage test scheme and a fast search

4 algorithm can reduce the complexity of BMGT to a bearable level Fast vertex search algorithm In order to reduce the huge amount of quadrilateral matchings without seriously aecting motion compensation accuracy, we designed a two stage search scheme by considering that object's motion can be decomposed into two components, translation and rotation. First each block is motion compensated for the translational component, then if the result is not satisfactory it is compensated for the rotational component of the motion. Hence the computational complexity is largely reduced as many blocks are motion compensated in the rst stage. Another important issue for computational complexity reduction is utilisation of fast search algorithms (logarithmic step) for block mapping. For blocks requiring BMGT, these methods can drastically reduce the number of search points. In this technique, called Fast Vertex Search (FVS) approach using fast search algorithms, the appropriate displacement for all the vertices of the quadrilateral in the previous frame are searched combinatorially. Several fast search algorithms have been successfully used, such as: Three Step Search (TSS) [9], Cross Search Algorithm (CSA) [10] or Orthogonal Search Algorithm (OSA) [11]. At each stage of the fast search algorithm, all the possible candidates are evaluated before proceeding to the next stage. The maximum number of combinations that can be reached by each method for!=16 pixel/frame is given in Tab. 1. The M No. of Op. FS (2! + 1) 8 1: TSS (1 + 8) 4 log 2! 2: CSA (1 + 4) 4 (log 2! + 1) 2: OSA (1 + 2) 4 2 log 2! 648 Table 1. Computational complexity. number of operations (No. of Op.) for logarithmic algorithms (M) can also be determined regarding the number of stages (s) and the number of search points in each stage (n). The number of quadrilateral mappings required is given by Eqn. 7. M LOG = n 4 s: (7) Equation 7 shows that the number of quadrilateral mappings required manifestly depends upon the number of search points, n, in each step of the rst search algorithm (power of 4) D D/ Figure 2. OSA searching for w = 8 pixel/frame. Despite representing an optimal solution, the non-logarithmic Full Search method is not a practical solution. Simulations comparing TSS against OSA methods showed that TSS obtains slightly better motion compensation performances. However, the number of computations is dramatically reduced when OSA is used. Hence, similar to the results obtained in [12], among the three fast search methods the one which has the smallest n, i.e. (OSA), is preferred. Using OSA, as shown in Fig. 2, the overall computational complexity of BMGT can be obtained by summing the number of arithmetic operations required for each operation of BMGT. The overall computational load of BMGT for a motion speed of! = 16 pixels/frame and B = 16 pixel blocks is approximately D/ A M: In comparison BMA with the same search window size of! = 16 pixel/frame and the full search needs approximately 1089(511A + 2M) = A M: That is, the number of arithmetic additions is almost doubled, but the number of multiplications is 165 times larger. Note that due to the two-stage approach, many blocks may just be compensated with BMA, thus reducing the computational complexity. Moreover, if BMGT is combined with image segmentation, then smaller blocks need smaller search ranges, further reducing the computational load. For example, for 8 8 pixel blocks, the maximum motion speed becomes only! = 4 pixel/frame, which signicantly reduces the computational eort. 7 12

5 3. Image segmentation and bit rate generation Quadtree decomposition of blocks, [13], can be used eciently in combination with motion estimation, since larger blocks can be used when a large image area moves at the same direction and smaller blocks can be used to represent areas with more complex motion. Since higher compression ratios are achieved with larger blocks, blocks initially should be tested with a larger size, and segmentation can be terminated if the motion compensation is acceptable. In our method the top-down construction of the quadtree has three levels and starts with pixel blocks in the rst level, which is the maximum block size for partitioning a pixels image into nonoverlapping blocks. Transition between levels is conditioned to a test that evaluates the block representation at this level, and if the test is negative the segmentation continues. As mentioned earlier, the motion estimation is carried out in two stages. In the rst stage, if the result of BMA is satisfactory then the estimation and also segmentation terminates, otherwise BMST is used. This strategy not only reduces the computational complexity of the motion estimator, it also reduces the motion overhead information. This is because BMA uses only one vector (d x ; d y ) to represent the displacement of a block, while BMGT needs one vector for each vertex, i.e. four vectors per block. Further compression is achieved by variable length coding of the motion vectors Adjacent quadrilateral vertex approximation Since BMGT blocks require high overhead, ecient coding of these blocks for low bit rate video applications is highly desirable. The fact is that for a smooth moving object, all the four corners of a coded quadrilateral do not necessarily differ from their neighbouring ones. This is specially true for a rigid body movement. Therefore, BMGT blocks can be eciently coded if those corners of the neighbouring quadrilaterals, which have common coordinates, are represented by one of them. For corners which do not exactly coincide with each other, still their coordinates can be approximated, provided that motion compensated distortion due to the resulting approximation is not large. As can be seen in Fig. 3, two vertices in B and three vertices in D may be approxim- D A Figure 3. Vertex approximation. ated if the introduced distortion is minimum. In C all four vertices are coincident, thus only one pair of motion vectors is required to represent the four vertices that belong to four adjacent blocks. Figure 4 shows the improvement in the bit rate, Bit/frame Frame No. C B Sergio sequence (12.5Hz) Approximated vertices Indep. coded vertices Figure 4. Bit rate performance for dierent bit generation techniques. when the close corners are approximated. The average bit rate is reduced from 5526 kbit/frame to 4843 kbit/frame and the respective average PSNR also reduces from 36:3 db to 35:9 db. 4. Experimental results To test the eciency of the proposed method in tracking and compensating for complex motion, a moving head and shoulders sequence with mainly rotational motion was recorded. The sequence, Sergio, was recorded as 25 frames/s interlaced sequence at broadcast quality. The speed of the head movement is such that it takes almost 3 seconds (75 frames) to move from one side to the other. The sequence was then both temporally and spatially sub-sampled at 12:5Hz, with pixels resolution. Finally, the subsampled sequence was coded at approximately 5 kbit/s, using only motion vectors of both the BMGT and BMA methods. The maximum motion speed for the reference BMA was 8 pixels per frame, while that of BMGT was half the block size at each stage of segmentation. That is they were

6 BMA - Frame 10 BMGT - Frame 10 BMA - Frame 20 BMGT - Frame 20 BMA - Frame 30 BMGT - Frame 30 BMGT - Frame 29 BMGT - Frame 30 BMGT - Frame 31 BMGT - Frame 32

7 16, 8 and 4 pixels per frame for 32 32, and 8 8, respectively. Pictures BMGT and BMA from Frame 10 to 30 and shows samples (every tenth coded frame) of the reconstructed picture sequence, with the assumption that the rst frame is available at the receiver. As can be seen in this gure, the BMGT very accurately tracks the motion of the head, in particular detailed motion of opening and closing eyes and mouth. This is completely distorted in the BMA method. The ability of the BMGT as an ecient motion compensation extrapolator is due to combined transformation, segmentation and interpolation of the proposed method. Transformation and interpolation make it possible to extrapolate the current blocks of pixels from those of the previously deformed blocks. This can be veri- ed from the BMGT pictures, where four pictures frames close to the frame 30 are shown. Frame preceding frame 30, i.e. frame 29 and those following frame 30, i.e. frames 31 and 32, clearly show how the picture is extrapolated. This is specially evident in the area of the reconstruction of hair around the left ear. The segmentation makes it possible to encode such a picture at around 5 kbit/frame. Such a good motion tracking ability, as well as the opening and closing of eyes is due to the vertex independent nature of BMGT, where all the four corners of the quadrilaterals are free to move in any direction. This is in contrast to the continuous mesh methods where the neighbouring vertices are interdependent, reducing the exibility of the transformation. For example, consider that just only one pixel in the previous frame has a pixel intensity close to the average intensity of the pixels in the current frame (Fig. 5). Now if all the four vertices of the quadrilateral move Figure 5. a) Four-to-one pixel mapping. b) Oneto-four pixel mapping. towards the centre by N 2 pixels, then that pixel alone is matched to the current block, that is all the pixels of the current block are interpolated just from one pixel. Of course, the reconstructed picture will be heavily ltered [14]. Growing of the hair in frames of BMGT 29 to 32 is due to this kind of process, and similarly the opening and closing of eyes and mouth. This is the process used to compensate uncovered background, mapping blocks from regions in the previous frame with luminance similarities. Note that since only luminance information is used for motion tracking, moving coloured objects cannot necessarily be reproduced with the same delity. Because mapping of chrominance uses the same information as luminance, then some mappings may be incorrectly addressed, which needs a subsequent coding of this error. Figure 6 shows the peak signal-to-noise ratio (PSNR) of the reconstructed sequence for three techniques used: BMGT, BMA-Quadtree and BMA. Both, BMGT and BMA-Quadtree, use a PSNR(dB) Sergio sequence (12.5Hz) BMST BMA-Quadtree BMA-Fixed size FRAME No Figure 6. PSNR performance of the spatial transformation in motion compensation. block segmentation algorithm yielding block sizes between pixels and 8 8 pixels. Also BMGT includes BMA coded blocks. These two algorithms also create a bit rate of approximately 5 kbit/frame. On the other hand, BMA uses a xed block size of pixels and its motion vectors are encoded at approximately 2 kbit/frame. The improvement in PSNR achieved by BMGT correlates well with the subjective picture quality comparison of the pictures. There are two major reasons for this improvement: the eciency of the quadtree segmentation, and the use of geometric transformations for block matching. Comparing BMA-Quadtree against BMA, we can see that the improvement achieved by quadtree segmentation has a signicant cost in the bit rate, which increases approximately 3 kbit/frame. But, with the same bit rate and the same range size blocks, geometric transformations further improve the motion compensation, approximately 2 db compared against the BMA-Quadtree. In BMGT,

8 due to the properties of bilinear transformation, most of the complex motion is compensated with large blocks, not needing the quadtree segmentation. On the other hand, in BMA-Quadtree, when the motion is not translational, the block is quadtree segmented in order to approximate a large block with complex motion by translation components of small blocks. Frequently, this approximation is not achieved properly, but as the PSNR distortion may be smaller a false motion compensation is done. It also may happen with BMGT, but because this technique is capable of compensating a wider range of motion than BMA technique, the number of false motion vectors is reduced. Being very ecient for tracking of complex motion, the bilinear transform also compensates uncovered background, scaling of objects in the scene and reconstruction of nonexistent shapes in the previous frame. However, this only happens if there is any shape with similar luminance in the previous frame. 5. Summary It was shown that motion estimation with geometric transformation can be eciently employed in interframe codecs to compensate complex motion. The mapping function matches itself very well to non-uniform changes between video frames, such that motion vectors alone can be sucient to code a head-and-shoulders video sequence at very low bit rate. Due to the nature of transformation where quadrilaterals have to be matched, extrapolation is needed. Such extrapolation provides an opportunity whereby, without increasing the overhead, motion compensation with maximum possible degree of non-integer pixel estimation accuracy is achievable. However, this sophisticated motion estimation has high computational complexity. It was shown that using scanline techniques and logarithmic step search algorithms, the computational complexity is enormously reduced, to a degree which is aordable for practical implementation. References [1] A. Netravali and B. Haskell, Digital Pictures. Representation and compression. New York: Plenum Press, [2] A. Netravali and J. O. Limb, \Picture coding: A review," IEEE Proceedings, vol. 68, pp. 366{406, March [3] H. Mussmann, P.Pirsch, and H. Garllet, \Advances in picture coding," IEEE Proceedings, vol. 73, pp. 631{670, April [4] Y. Nakaya and H. Harashima, \Motion compensation based on spatial transformations," IEEE Transactions on Circuits and Systems for Video Technology, vol. 4, pp. 339{356, June [5] V. Seferidis and M. Ghanbari, \General approach to block-matching motion estimation," Journal of Optical Engineering, vol. 32, pp. 1464{1474, July [6] G. J. Sullivan and R. L. Baker, \Motion compensation for video compression using control grid interpolation," in IEEE International Conference on ASSP, pp. 2713{2716, [7] J. Jain and A. Jain, \Displacement measurement and its application in interframe image coding," IEEE Transactions on Communications, vol. COM-29, no. 12, pp. 1799{1808, [8] G. Wolberg, Digital Image Warping. Los Alamitos, USA: IEEE Computer Society Press, [9] T. Koga, K. Iinuma, A. Hirano, Y. Iijima, and T. Ishiguro, \Motion compensated interframe coding for video conferencing," in Proceedings Nat. Telecommunications Conference, (New Orleans, USA), pp. G5.3.1{ 5.3.5, December [10] M. Ghanbari, \The cross-search algorithm for motion estimation," IEEE Transactions on Communications, vol. COM-38, pp. 950{ 953, [11] A. Puri, H. M. Hang, and D. L. Schlling, \An ecient block-matching algorithm for motion compensated coding," in Proceedings of the ICASSP 87 Conference, pp { , [12] I. N. G. K. T. Tan and M. Ghanbari, \Fast motion estimation with spatial transformation," Electronics Letters, vol. 30, pp. 847{ 849, May [13] P. Strobach, \Tree-structured scene adaptive coder," IEEE Transactions Communications, vol. 38, pp. 477{486, April [14] B. Girod, \Motion-compensating prediction with fractional-pel accuracy," IEEE Transactions on Communications, vol. 41, pp. 604{ 612, April 1993.