AUDIO-VISUAL services over very low bit rate channels,

Transcription

1 1270 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 9, SEPTEMBER 1997 Occlusion-Adaptive, Content-Based Mesh Design and Forward Tracking Yucel Altunbasak and A. Murat Tekalp, Senior Member, IEEE Abstract Two-dimensional (2-D) mesh-based motion compensation preserves neighboring relations (through connectivity of the mesh) as well as allowing warping transformations between pairs of frames; thus, it effectively eliminates blocking artifacts that are common in motion compensation by block matching. However, available 2-D mesh models, whether uniform or nonuniform, enforce connectivity everywhere within a frame, which is clearly not suitable across occlusion boundaries. To this effect, we hereby propose an occlusion-adaptive forward-tracking mesh model, where connectivity of the mesh elements (patches) across covered and uncovered region boundaries are broken. This is achieved by allowing no node points within the background to be covered (BTBC) and refining the mesh structure within the model failure (MF) region(s) at each frame. The proposed content-based mesh structure enables better rendition of the motion (compared to a uniform or a hierarchical mesh), while tracking is necessary to avoid transmission of all node locations at each frame. Experimental results show successful motion compensation and tracking. Index Terms Mesh refinement, occlusion-adaptive forward tracking, 2-D content-based mesh design. I. INTRODUCTION AUDIO-VISUAL services over very low bit rate channels, such as the public switched telephone network (PSTN) and wireless media, is an important emerging application [1] [4]. The International Telecommunications Union (ITU- T) has recently adopted the draft recommendation H.263 [5] as a worldwide standard for video compression/decompression at less than 64 kb/s. Recommendation H.263 employs an improved version of the classical block-based motioncompensated discrete cosine transform (MC-DCT) (hybrid) coding method, which also forms the basis of the prior H.261, MPEG-1, and MPEG-2 standards. The MC-DCT coding strategy is based on the source model of 2-D translational blocks, which is known to result in blocking and mosquito artifacts at very low bit rates, especially with common intermediate format (CIF) or larger video formats. This is because translational block model i) cannot handle image rotations and scaling and ii) cannot preserve the neighboring relations between the blocks, resulting in blocking artifacts. Manuscript received December 26, 1995; revised January 20, This work was supported in part by a National Science Foundation IUCRC grant and a New York State Science and Technology Foundation grant to the Center for Electronic Imaging Systems, University of Rochester. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Eric Dubois. The authors are with the Department of Electrical Engineering and Center for Electronic Imaging Systems, University of Rochester, Rochester, NY USA ( altunbas@ee.rochester.edu). Publisher Item Identifier S (97) Generalized block matching [6] was proposed to allow for spatial transformations (e.g., affine, perspective, etc.) to overcome the limitations of the translational model. However, it does not address the discontinuity problem at the block boundaries. Overlapped block motion compensation, which is included in the advanced prediction mode of H.263, offers a partial solution to both problems [7]. A promising alternative is motion compensation by 2-D mesh models. 2-D meshes allow for spatial transformations and preserve neighboring relations between patches, while requiring transmission of only a coarsely sampled motion field (a dense motion field can be interpolated from vectors at the node locations). 2-D meshes can be classified as uniform (regular) meshes with equal size elements, and nonuniform (hierarchical or content-based) meshes that adapt to particular scene content. Nonuniform mesh modeling/design may be viewed as a method of irregular sampling of the dense motion field. Brusewitz [8] was among the first who proposed trianglebased motion compensation, where a regular triangular mesh is overlayed on the image. Sullivan and Baker [9] used quadrilateral meshes for motion compensation under the name control grid interpolation. Motion compensation within each mesh element (patch) is accomplished by means of a spatial transformation (affine, bilinear, etc.) whose parameters can be computed from node-point motion vectors. Recently, Nakaya et al. [10] proposed a hexagonal matching procedure for motion estimation based on a uniform mesh. Uniform meshes are suitable for motion compensation by redesign ; that is, a new uniform mesh is overlayed on each frame, and motion vectors at the node points are estimated from frame to for motion compensation. However, uniform meshes are often inadequate for motion rendition in the vicinity of object boundaries, where a patch may contain two or more different motions. This problem may be addressed by splitting those patches with more than one motion into smaller elements, resulting in a hierarchical mesh. Huang et al. [11] employed a hierarchical mesh based on a quadtree structure. Nonuniform mesh design using edge analysis was proposed in [12]. Clearly, information about mesh elements that are split must be transmitted as overhead. A more fundamental approach to overcome the problem of mesh elements with more than one motion is to design a content-based mesh, which is not limited to a pseudoregular hierarchical structure. Wang et al. [13] advanced an optimization framework for motion compensation based on an active mesh, which adapts to scene content. However, content-based meshes are not suitable for motion compensation by redesign, /97$ IEEE

2 ALTUNBASAK AND TEKALP: MESH DESIGN AND FORWARD TRACKING 1271 because transmission of all node locations at each frame constitute an excessive amount of overhead. Therefore, motion compensation by a content-based mesh must be coupled with a forward tracking procedure, where a new mesh is designed only at selected key frames, and, in between, it is tracked by node point motion vectors estimated from frame to. Such a scheme works well except around occlusion boundaries, where reliable forward tracking is not possible with the available methods. Note that an important drawback of all these mesh models (uniform, hierarchical or content based) is that they enforce connectivity of the structure everywhere. This is similar to imposing a global smoothness constraint on the 2-D motion field, which is clearly unsuitable across motion and occlusion boundaries. To this effect, we introduce the concept of occlusionadaptive mesh modeling. Occlusion regions, classified as background to be covered (BTBC) and uncovered background (UB), may appear at the object boundaries (due to global object motion) or within objects (due to local motion or deformations). The latter is generally referred to as selfocclusion. We hereby propose an occlusion-adaptive, contentbased mesh design and forward tracking procedure, where no node points are allowed in the BTBC regions and the mesh within the model failure region(s) is redefined for subsequent tracking of these regions. The success of forward tracking is closely related to how well we can detect occlusion and model failure regions and estimate the motion field in the vicinity of their boundaries. Some known occlusion detection methods include [14] [16]. In motion compensation by forward tracking, positions of all nodes need to be transmitted only in selected key frames; in all other frames, it suffices to transmit the boundaries of the BTBC regions (to determine nodes to be killed) and the locations of the newly born nodes. Thus, the proposed approach combines the benefit of a content-based mesh with a small amount of overhead transmission. The paper is organized as follows. Section II proposes a new, efficient, occlusion-adaptive, content-based mesh design procedure that is employed to design the initial mesh, and select new node points within the UB region for the subsequent frames. Section III covers node-point motion estimation for the purposes of motion compensation and mesh tracking, followed by a forward node tracking and mesh refinement algorithm. An occlusion-adaptive motion compensation algorithm by redesign is presented in the Appendix (for comparison of the results). Experimental results with frames of the mother and daughter sequence, provided in Section IV, show that the proposed occlusion-adaptive, content-based mesh design and tracking procedure is successful. II. OCCLUSION-ADAPTIVE MESH DESIGN This section starts with a discussion of the forward-tracking occlusion-adaptive mesh concept. We then provide an algorithm to determine the BTBC region(s) and approximation of their boundaries by polygons. Next, we present a practical method to design an occlusion-adaptive, content-based mesh with triangular patches, such that patch boundaries conform Fig. 1. Illustration of the occlusion adaptive mesh concept. with the boundaries of objects in the scene, BTBC and UB regions, and motions, as much as possible. A. The Occlusion-Adaptive Mesh Concept We introduce the occlusion-adaptive mesh concept to overcome a fundamental limitation of standard mesh models (which enforce continuity of motion across the whole frame) in dealing with multiple motion and occlusion regions. Let us consider the example of a single moving object against a still background, which is shown in Fig. 1. Suppose that the elliptical object is translating to the right, resulting in two types of motion boundaries: To the right is the motion boundary associated with the BTBC region, and to the left is the boundary associated with the UB region. Assuming that these boundaries and the motion vectors in the vicinity of them can be correctly estimated, the BTBC region in frame (marked in Fig. 1) should completely disappear (get covered) in frame, and node points in frame along the motion boundary associated with the UB region (left boundary in Fig. 1) should be split into two. The splitting is a result of the fact that these nodes belong to triangles covering both the background and the object; thus, they need to be assigned two different motion vectors (one to model the motion of the background and another for the motion of the object). An occlusion adaptive mesh should, therefore, have the following properties: i) no nodes are present within the BTBC region(s) in the temporally first frame since they will be covered in the next frame, hence no meaningful motion vectors can be computed for them; ii) nodes that are on object boundaries may be assigned more than one motion vector, since they are common to two or more objects; and iii) the mesh is redesigned within the UB region(s) in the temporally next frame for subsequent tracking of newly exposed regions. Clearly, accurate modeling of these regions using a mesh model requires placement of node points along them (polygon approximation of these regions). The success of this scheme, however, depends on how reliable the BTBC and UB regions, and the motion vectors in the vicinity of these regions can be estimated. Because, in practice, the BTBC and UB regions cannot be accurately estimated due to inaccuracy of motion vectors in the vicinity of these regions, in this paper we have chosen an implementation that favors robustness of the motion compensation in the presence of such estimation errors. To this effect,

3 1272 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 9, SEPTEMBER 1997 we estimate the model failure (MF) region instead of the UB region (see Section III-C). That is, our mesh design algorithm first estimates the BTBC region(s) in the temporally first frame, and does not place any nodes within them. Then, forward motion vectors at the node points are estimated, and affine mapping parameters are computed for each mesh element from these node point motion vectors. Motion compensation of the temporally next frame is performed by inverting each affine mapping. Finally, the MF region is estimated (in the temporally next frame) from the displaced frame difference as a function of the parametric motion field; the intensity within the MF region is intracoded, and the mesh within the MF region is refined for purposes of tracking these regions (e.g., newly appearing objects) in the subsequent frames. Ideally, as the estimated motion vectors approach the true motion vectors, the MF region approaches the UB region; therefore, our implementation is consistent with the above-cited properties of an occlusion-adaptive mesh. Robustness of our implementation is due to i) the estimated BTBC region corresponds to pixels where motion vectors are unreliable (thus, placing nodes in these regions should be avoided); and ii) the estimated MF regions correspond to pixels where resulting motion affine compensation is erroneous (thus, mesh refinement within such regions improves motion compensation accuracy for future frames). B. BTBC Region Detection The BTBC region(s) refers to clusters of pixels within the temporally first frame, which are covered in the temporally next frame (see Fig. 1). The boundaries of the BTBC region(s) can be estimated by thresholding the displaced frame difference, computed from the forward dense motion field estimated from frame to. Clearly, the accuracy of the occlusion boundaries depends on the accuracy of dense motion estimates, especially in the vicinity of the occlusion regions. In this work, we have used optic-flow equation based methods, such as the method of Lucas Kanade [20] and Horn Schunck [21], since they yield smoother (hence closer to the actual) motion fields than, for example, block matching. It should be noted that values of the smoothness and/or blurring parameters in these methods are important, since, if the motion field is oversmoothed, motion tracking over small regions becomes impossible. For the sake of minimizing the computational burden, 2-D dense motion estimation and occlusion detection are performed only over the changed regions (which can be easily detected from the frame difference). A summary of the BTBC detection algorithm is as follows. 1) Calculate the frame difference (FD); and determine the change detection mask (CDM) by thresholding and postprocessing as follows: a) apply median filtering with a 5 5 kernel; b) apply three successive morphological closing operations with a 3 3 kernel, followed by three morphological opening operations with the same kernel; c) eliminate small regions which are smaller than a predetermined size. Fig. 2. Approximation of the occlusion region by a polygon. 2) Estimate the dense motion field from frame to frame, within the CDM (e.g., using the method of Lucas Kanade [20] or Horn Schunck [21]). 3) Motion compensate frame from frame to compute the predicted estimate using the estimated dense motion field. 4) If the displaced frame difference, DFD, where is the observed intensity, is greater than a predefined threshold, then label as an occlusion pixel. 5) Perform postprocessing to form smooth BTBC region(s) as in Step 1). The thresholding operation may result in small pixel clusters. Postprocessing is applied following thresholding to form contiguous CDM and BTBC regions. Median and morphological filtering operations remove small pixel clusters, and also fill in small holes. The sizes of the small clusters to be eliminated and holes to be filled in depend on the filter kernel size. Note that in the case of backward motion compensation, the roles of the BTBC and UB regions are interchanged. Next, the boundaries of these regions are approximated by polygons to facilitate occlusion-adaptive mesh design. C. Polygon Approximation The boundary of the occlusion regions (BTBC and UB) needs to be approximated by a shape model that can be represented with a few parameters. Most commonly employed shape models are polygonal and/or B-spline approximations. Here, we employ a polygonal approximation (see Fig. 2), because of its simplicity and robustness. Furthermore, a polygonal boundary naturally fits with the boundaries of the proposed occlusion-adaptive mesh model. We use a polygon approximation algorithm, which is similar to that proposed by Gerken [18]. It is summarized below. 1) Find a pair of pixels, among all possible pairs, on the boundary of the occlusion region, which has the maximum distance between them. The line drawn between these two points (vertices 1 and 2) is called the main axis. 2) Find the two points on the boundary with the largest perpendicular distance from the main axis to the left and right of the axis (vertices 3 and 4). These four vertices are called the initial vertices. 3) Draw a straight line from vertex 1 to the nearest vertex (clockwise). If, for every point on the boundary between

4 ALTUNBASAK AND TEKALP: MESH DESIGN AND FORWARD TRACKING 1273 these two vertices, the maximum distance between the straight line and the boundary is below a certain threshold and the area between the boundary and the straight line is less than 5% of the area of the entire occlusion region, then no new vertices need to be inserted on this segment of the boundary. If, however, both criteria are not satisfied, a new vertex is inserted at the pixel with the maximum distance from the straight line, and this procedure is repeated until no new nodes are needed within each new pair of boundary segments. 4) Repeat 3) for each of the three remaining segments between the initial vertices. The nodes of the polygon approximations will serve as nodal points of the 2-D mesh (see mesh refinement algorithm in Section III-B) to be able to accurately track the motion in the vicinity of occlusion regions by means of motion vectors at these node points. D. Content-Based Mesh Design Prior work on adaptive mesh design includes optimization and split-and-merge methods. Designing an optimal mesh structure requires global optimization of a suitable cost function [13]. However, most practical optimization methods converge to a local minimum which is close to a uniform mesh (assuming the initial mesh is a uniform one). Split-and-merge methods successively divide, starting possibly with a uniform mesh, those patches that do not satisfy a predetermined criterion in an attempt to find locally optimum solutions [11], [17]. Some of the inserted nodes may need to be treated specially to preserve the connectivity (geometry) of the mesh and/or children-parent patch relationships in hierarchical mesh refinement procedures [11]. Here, we propose a new, computationally efficient algorithm for content-based 2-D triangular mesh design. The algorithm consists of a nonuniformly spaced node-point selection procedure, followed by triangulation using the selected node points. The basic principle of the algorithm is to place node points in such a way that mesh boundaries align with object boundaries and the density of node points is proportional to the local motion activity. The former is attempted by placing node points on spatial edges (pixels with high spatial gradient). The latter is achieved by allocating node points in such a way that a predefined function of the displaced frame difference (DFD) within each patch attains approximately the same value. An outline of the algorithm is as follows. 1) Label all pixels, except those in the BTBC polygon(s), as unmarked. Include all corner points of the BTBC polygon(s) (see Section II-C) in the list of selected node points. 2) Compute the average displaced frame difference DFD given by DFD DFD (1) where DFD stands for the displaced frame difference as a function of the dense motion field computed in the BTBC region detection procedure above, the Fig. 3. Demonstration of proximity constraints in the mesh design algorithm. summation is over all unmarked points, is the number of unmarked pixels, and is a positive number (e.g., ). 3) For each unmarked pixel, compute a cost function where and stand for the partials of the intensity with respect to and coordinates evaluated at the pixel. The cost function is related to the spatial intensity gradient so that selected node points; hence, the boundaries of the patches, coincide with spatial edges. 4) Find the unmarked pixel with the highest, which is not closer to any other previously selected node point than a prespecified distance. Label this point as a node point. 5) Grow a circle about this node point until DFD in this circle is greater than DFD. Label all pixels within the circle (depicted in Fig. 3) as marked. 6) Go to 2) until a desired number of node points,, are selected, or the distance criterion in 4) is violated. 7) Given the selected node points, apply a triangulation procedure (e.g., Delauney triangulation [19]) to obtain a content-based mesh. Steps 1 6 of this procedure constitute the node-point selection algorithm. A flowchart of the above mesh design procedure is shown in Fig. 4. Node points are placed at pixels with the highest spatial gradient among all eligible (unmarked) points, so that they correspond to spatial edge points. In reference to Fig. 3, a small circle indicates a high temporal activity, while a large circle indicates low temporal activity. The pixels within each circle are marked, so that another node point cannot be placed within a marked circle. This process controls the node point density in proportion to the local temporal activity (motion). (2)

5 1274 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 9, SEPTEMBER 1997 Fig. 5. Illustration of inconsistent motion vectors. Fig. 4. A flowchart of the content-based mesh design algorithm. III. FORWARD TRACKING This section presents methods for i) node-point motion estimation, ii) affine motion compensation, iii) model failure region detection, and iv) forward mesh tracking. The forward motion compensation algorithm aims to achieve the goals that the BTBC region(s) in frame should disappear (get covered) in frame, and the MF region(s) in frame should reduce to the UB region(s). A. Node-Point Motion Computation Motion compensation using a 2-D mesh requires computation of the parameters of a spatial transformation within each triangular mesh element (patch). It is well known that the parameters of an affine mapping can be uniquely estimated from three point correspondences (e.g., at the vertices of a triangular patch). Therefore, in order to estimate the affine mapping parameters for motion compensation of each patch, it suffices to estimate the motion vectors at the node points. There are two approaches for node point motion estimation: i) estimate motion vectors at the node points independently of each other (e.g., perform block matching at each node point or sample motion vectors from a dense optical flow field); ii) estimate motion vectors optimized for warping transformations of each triangular patch (e.g., using a constrained search [10], [23] or closed-form solutions [24]). Because the latter option is computationally more intensive, this paper employs the former. That is, we sample (pick) our node point motion vectors from the dense motion field, which is already estimated during the BTBC region detection/meshdesign procedures. An optical flow-based method has been preferred in Section II-B over block matching, because opticalflow methods yield smoother dense motion fields, which are more suitable for parameterization [22]. Although hierarchical block matching for dense motion estimation yields high motion-compensation peak signal-to-noise ratio (PSNR), the motion field usually contains several outliers (because of the aperture problem and lack of explicit smoothness constraints across the blocks). It should be noted that the value of the smoothness parameter in [20] or [21] is important, since if the motion field is oversmoothed tracking of motion over small regions becomes impossible. It is possible that motion vectors sampled from a dense motion field are inconsistent in the sense that they do not preserve the connectivity of the mesh structure. This is illustrated in Fig. 5, where the node in frame is connected with the nodes and. The motion vector at the node must move it to inside the polygon formed by the motioncompensated nodes and in frame. However, because the dense motion estimation method utilized to compute the node-point motion vector at the node does not employ such a constraint, this condition may be violated as shown in Fig. 5. When such a condition occurs, the estimated motion vector should be replaced by a motion vector that is interpolated from those of the surrounding nodes, or alternatively, a local search may be conducted to estimate the motion vectors at such nodes. Therefore, we propose a motion vector postprocessing algorithm to preserve the connectivity of the patches. This is achieved by ordering the sampled motion vectors according to a measure of confidence (as a function of the resulting local displaced frame difference). In the case of a motion vector cross over, the lowest confidence motion vectors are interpolated from the higher confidence ones to eliminate cross-overs as described in the following. 1) Determine the ordering of the nodes for postprocessing as follows. a) For each node, find its enclosing polygon (determined by nodes connected to node ). b) Calculate the following criterion function DFD (3) where and are positive scalars, DFD, and denote the displaced frame difference, variance, and number of points in the polygon enclosing the node, respectively.

6 ALTUNBASAK AND TEKALP: MESH DESIGN AND FORWARD TRACKING 1275 c) The node with the highest is then processed first. Prioritization enables resolution of conflicts at nodes with lowest confidence motion vectors first in favor of those nodes with more reliable motion vectors. 2) Scan the nodes in the order determined in Step 1 to detect nodes with inconsistent motion vectors as follows: At each node, a) Find all the nodes connected to node, and label them as,, where is the number of nodes connected to. b) Find the motion-compensated node locations, in frame using the motion vectors at the nodes. Form the polygon defined by. c) Motion compensate the node to find.if is inside the polygon, go to next node in the order. Otherwise, go to 3) 3) Interpolate the motion of the node from its neighbors as follows: Fig. 6. Affine motion compensation. 1) For each patch at frame, the affine mapping parameters can be computed by solving the linear equation (8) where is the node-motion vector at the node, and is the distance between the node and node. Postprocessing provides increased robustness to errors in motion estimation around occlusion boundaries. The affine mapping parameters for each triangular patch are then computed from the postprocessed, sampled node-point motion vectors at the respective vertices. B. Motion Compensation Node point motion vectors establish a set of point correspondences from frame to frame, which are used to determine a set of backward affine spatial transformation from frame to (see Fig. 6), given by (4) (5) (6) (7) where denotes the coordinates of a pixel in frame which corresponds to point in frame. Note that, since node point motion vectors are computed from frame to, the coordinates are real numbers that may not correspond to pixel locations. The computation of the set of affine parameters and affine motion compensation can be described through the following procedure. where and are the corresponding node positions. 2) All pixels within the current patch at frame are motion compensated from frame by using the affine mapping (7). If the corresponding location in frame is not a pixel location, then bilinear interpolation is used. C. Model Failure Region Detection Following motion compensation by the parametric motion field, the MF region is detected by thresholding the displaced frame difference where denotes the motion compensated frame using the affine motion field computed in Section III-B, and is the MF region detection threshold. Small pixel clusters and small holes are removed using the same postprocessing steps as discussed in BTBC region detection (see Section II-B). The MF regions correspond to clusters of pixels where affine motion compensation fails. This may be due to presence of uncovered background (UB) regions, inaccuracy of node point motion estimation, or insufficiency of the affine motion model. (Note that the UB region(s) refer to pixels in the temporally next frame (frame ), which are uncovered as a result of the interframe motion [see Fig. 1)]. Hence, it is expected that with successful tracking, the MF regions should reduce to the UB regions. D. Mesh Refinement The simplest scheme for forward tracking of the mesh from frame to is to propagate all nodes by their motion vectors. However, since motion compensation is deemed unsuccessful within the MF region, nodes that map into the MF (9)

7 1276 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 9, SEPTEMBER 1997 Fig. 7. Addition, deletion, and propagation of nodes for mesh refinement. region in frame cannot be considered reliable. Thus, we model the boundary of the MF region with a polygon (similar to the procedure given in Section II-C) and delete all nodes that are mapped into the MF polygon in frame. Note that the coordinates of the corners of the boundary polygon is sufficient to describe all nodes that should be deleted. It is assumed that the contents of the MF polygon (intensity values within the MF polygon) at each frame have to be transmitted once as overhead information. Finally, the mesh tracked from frame is refined by redesigning a new mesh within the MF region (excluding the BTBC region of the next frame) for the purposes of tracking the contents of the MF polygon in the subsequent frames. Mesh refinement procedure in frame does not place any nodes in the BTBC region between the frames and, since this region will vanish (get covered) in frame (just as the initial mesh design procedure does not place any node points in the BTBC region in frame ). The regions within which nodes are added, deleted, and propagated are illustrated in Fig. 7. Note that corners of the BTBC polygon (between frames and ) are accepted as nodes for mesh refinement. Additional nodes within the BTBC region may also be included as determined by the mesh design procedure in Section II-D. E. The Complete Forward-Tracking Algorithm This section summarizes the complete forward-tracking mesh algorithm, including the BTBC region detection, mesh design, node-point motion estimation, motion compensation, MF region detection, and mesh refinement steps, as follows. 1) Set. Find the BTBC in the th frame given the dense motion field from the th to st frame (see Sections II-B and II-C). 2) Design a 2-D content-based mesh for the th frame, where no nodes are allowed in the BTBC (see Section II-D). (a) (b) Fig. 8. Flowchart of motion compensation/tracking algorithms. (a) Motion compensation by forward tracking algorithm. (b) Motion compensation by redesign algorithm. 3) Compute node-point motion vectors as described in Sections III-A, and perform motion compensation as in Section III-B. 4) Propagate all node points by their computed motion vectors. Increment by one. Find all MF regions using the actual and reconstructed th frames (as described in Section III-C). Find the enclosing polygons for each MF region. Delete all node points inside the enclosing polygons. 5) Compute the dense motion field from the th to st frame, and find the BTBC in the th frame. Employ the following mesh refinement algorithm in the MF region (excluding the BTBC). a) Insert the corner points of the MF polygon in the new node list. b) Apply the node-point selection algorithm (see Section II-D) within the MF region (excluding the BTBC region). c) Reapply triangulation. d) Go to Step 3. A flowchart of the forward tracking algorithm is shown in Fig. 8(a). IV. RESULTS Experimental results are provided to compare the performance of the proposed motion compensation by occlusionadaptive forward mesh tracking versus two benchmark methods over frames of the MPEG4 mother and daughter test sequence. The sequence starts with slow movements of the head of the mother. However, the part between frames 56 and

8 ALTUNBASAK AND TEKALP: MESH DESIGN AND FORWARD TRACKING 1277 (a) (b) Fig. 9. Initial mesh structures. (a) Uniform mesh. (b) Content-based mesh overlaid on the first frame. 87 is especially challenging and well suited to demonstrate the occlusion-adaptive mesh tracking concept, since the mother s hand enters the field of view (covering the face of the daughter) generating significant BTBC and UB regions. The processed video is SIF format ( ), which is converted from the original ITU-R 601 format by using the subsampling filter specified in the document [25]. We identified two benchmark results: motion compensation by redesigning 1) a new uniform mesh at each frame, and 2) a new content-based mesh at each frame. An algorithmic description of the motion-compensation by redesign approach is given in the Appendix. The case of redesigning a uniform mesh is expected to be a lower bound on the performance of motion compensation by the proposed forward tracking content-based mesh, since the structure of the mesh may not fit the motion boundaries well, leading to multiple motions within a single patch. However, it requires transmission of no overhead information about the mesh structure. The case of redesigning a content-based mesh at each frame, on the other hand, is expected to be an upper-bound, since the mesh structure should fit scene content well at each frame; however, it requires transmission of all node locations at each frame. The proposed forward tracking content-based mesh is a compromise between these two, since it yields a mesh structure that fits the scene content without too much overhead transmission. To this effect, the efficacy of the proposed method has been evaluated based on how it compares against these benchmarks in terms of motion-compensation PSNR (MC-PSNR) and the number of node points whose coordinates need to be transmitted at each frame. The MC-PSNR values refer to the prediction PSNR of each frame based on the original of the previous frame, using the affine motion field interpolated from the node-point motion vectors. An initial content-based mesh with 200 nodes is designed for the first frame of the sequence, and tracked by the proposed algorithm. The initial uniform and content-based mesh models, overlaid on the first frame, are depicted in Fig. 9(a) and (b), respectively. MC-PSNR versus frame number obtained by the forward tracking algorithm is plotted in Fig. 10. The variation of the number of nodes versus frame number (governed by the mesh refinement algorithm) as well as the number of nodes whose coordinates need to be transmitted (as overhead) at each frame are plotted in Fig. 11. The fluctuation in the number of nodes used at each frame is due to the fact that the number of added nodes may be different than the deleted nodes. The average number of nodes used over 100 frames is approximately 214. Hence, approximately the same number of nodes is used in the benchmark experiments. That is, in the case of redesigning a uniform mesh, a regular grid of nodes are used at each frame [see Fig. 9(a)]; and in the case of redesigning a content-based mesh, 214 nodes are employed at each frame. The MC-PSNR values for each frame by the two redesign approaches are also plotted in Fig. 10. In all of these experiments, the same dense motion field, estimated by the method of Lucas Kanade implemented in three steps by using 11 11, 9 9, and 7 7 square blocks, respectively, has been used. For the purpose of spatio-temporal gradient estimation, each frame is blurred by using a 5 5 Gaussian blur kernel with variance 2.5 in each direction. (Recall that BTBC detection requires dense motion estimates; furthermore, the node-point motion vectors are sampled from this dense motion field.) The BTBC detection threshold is set equal to eight; the MF threshold is nine. Node-point motion vectors have been quantized with 0.25 pixel accuracy. The tracking performance of the mesh over frames is demonstrated in Fig. 12(e) (h). The detected BTBC region (on frame 59) and MF region (on frame 60) are depicted in Fig. 13(a) and (b), respectively. Inspection of Fig. 10 shows that the occlusion-adaptive forward mesh tracking algorithm is almost as good as redesigning a content-based mesh for each frame; and it is clearly superior to redesigning a uniform mesh at each frame. The average MC- PSNR over 100 frames was computed as 39.88, 39.67, and db for the cases of content-based redesign, occlusionadaptive forward tracking, and uniform redesign methods, respectively. This is a very encouraging result considering that the content-based redesign method requires transmission of all (214) node coordinates, whereas the tracking method only needs to transmit the boundary nodes of the BTBC regions (it suffices to describe the nodes to be deleted) and the coordinates of the nodes to be added. The average (over 100 frames) number of nodes whose coordinates need to be transmitted has been 11 for frames of the mother

9 1278 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 9, SEPTEMBER 1997 Fig. 10. Motion-compensation PSNR versus frame number for uniform mesh redesigned at each frame (uniform mesh); forward-tracking occlusion-adaptive content-based mesh (occlusion-adaptive tracking); and content-based mesh redesigned at each frame (redesign). Fig. 11. Number of nodes whose coordinates need to be transmitted; and total nodes versus frame number in the case of forward-tracking occlusion-adaptive content-based mesh. and daughter sequence (see Fig. 11). Whereas redesigning a uniform mesh at each frame does not require any overhead transmission, its PSNR is well below (approximately 2 db) that of motion compensation by the forward-tracking mesh. That is, a significant improvement in PSNR can be achieved by sending an overhead that does not exceed a few hundred bits per frame. V. DISCUSSION AND CONCLUSIONS Forward tracking mesh-based motion compensation serves more than a mere tool for improving compression efficiency. It also enables such functionalities as nonlinear video editing for special effects, image registration for augmented reality, synthetic-natural hybrid coding, etc. [23]. The proposed occlusion-adaptive, content-based mesh design and forward tracking algorithm has several desirable characteristics described below. Robustness against occlusion boundary and motion estimation errors: Accurate estimation of occlusion boundaries is a challenging task, since motion estimates around occlusion regions are generally inaccurate. The proposed scheme provides the desired robustness against such errors by redesigning the mesh within the MF region at each frame. The efficacy of the resulting tracking algorithm can be assessed by how close the MC-PSNR values at each frame comes to that of redesigning a content-based mesh at each frame. Extensibility for object scalability: Object-scalable mesh design and tracking introduces the additional consideration that no patch should cross over the object boundaries in any frame (which imposes a constraint on the mesh design and tracking algorithms). The proposed scheme can be easily extended for object scalability although that is not addressed in this paper. The proposed method requires a forward dense motion field (within the changed region) for BTBC region estimation. If the estimation of a full dense motion field cannot be implemented within the processing requirements of the application, it may be interpolated from a subsampled motion field estimate; hence, trading accuracy with speed. The density of subsampled motion estimation may be adapted to the computational complexity requirements of the application. Finally, the proposed occlusion-adaptive forward mesh tracking algorithm can be extended to a complete 2-D mesh-

10 ALTUNBASAK AND TEKALP: MESH DESIGN AND FORWARD TRACKING 1279 (a) (e) (b) (f) (c) (g) Fig. 12. (d) (h) Comparison of forward mesh tracking and redesign on frames 58 61: (a) (d) tracking; (e) (h) redesign. based codec by adding a model failure (intra) coding module, node point and motion vector coding modules, and bit budget control module including node number control scheme. This is currently under investigation.

11 1280 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 9, SEPTEMBER 1997 REFERENCES (a) (b) Fig. 13. (a) BTBC region in frame 59. (b) MF region in frame 60. APPENDIX MOTION COMPENSATION BY REDESIGN Here we present a forward motion-compensation scheme, where a new mesh is designed between each pair of frames for comparison purposes. This method does not involve tracking; thus, it can be used as a benchmark to evaluate the performance of the proposed forward-tracking algorithm. Note that the redesign method can be used for either forward or backward motion compensation. The algorithm for the case of forward motion compensation is summarized below. 1) Set. 2) Estimate the dense motion field from the th to st frame, and find the BTBC region in the th frame (see Sections II-B and II-C). 3) Design a mesh (uniform or content based) for the th frame. 4) Estimate node-point motion vectors as described in Section III-A, and perform motion compensation (see Section III-B). 5) Increment by 1. Go to Step 2). A flowchart of this algorithm is shown in Fig. 8(b). A uniform mesh is obtained by segmenting the frame into equal size triangular elements; hence, there is no overhead to transmit the mesh structure. Note that in case a content-based mesh is employed, no nodes are allowed within the BTBC region (see Section II-D), and all node locations need to be transmitted at each frame. [1] H. G. Musmann, M. Hotter, and J. Osterman, Object oriented analysissynthesis coding of moving images, Signal Proc.: Image Commun., vol. 1, pp , Oct [2] H. Li, A. Lundmark, and R. Forchheimer, Image sequence coding at very low bit-rates: A review, IEEE Trans. Image Processing, vol. 3, pp , Sept [3] K. Aizawa and T. S. Huang, Model-based image coding: Advanced video coding techniques for very low bit-rate applications, Proc. IEEE, vol. 83, pp , Feb [4] T. Ebrahimi, E. Reusens, and W. Li, New trends in very low bit-rate video coding, Proc. IEEE, vol. 83, pp , June [5] ITU-T Draft Recommend. H.263, Video coding for low bit-rate communication (TMN5), July [6] V. Seferidis and M. Ghanbari, General approach to block-matching motion estimation, Opt. Eng., vol. 32, pp , July [7] M. Orchard and G. Sullivan, Overlapped block motion compensation: An estimation-theoretic approach, IEEE Trans. Image Processing, vol. 3, pp , Sept [8] H. Brusewitz, Motion compensation with triangles, in Proc. 3rd Int. Conf. 64-kbit Coding of Moving Video, Rotterdam, The Netherlands, Sept [9] G. J. Sullivan and R. L. Baker, Motion compensation for video compression using control grid interpolation, in Proc. ICASSP 91, Toronto, Canada, pp [10] Y. Nakaya and H. Harashima, Motion compensation based on spatial transformations, IEEE Trans. Circuits Syst. Video Technol., vol. 4, pp , June [11] C. L. Huang and C. Y. Hsu, A new motion compensation method for image sequence coding using hierarchical grid interpolation, IEEE Trans. Circuits Syst. Video Technol., vol. 4, pp , [12] J. Nieweglowski, T. G. Campbell, and P. Haavisto, A novel video coding scheme based on temporal prediction using digital image warping, IEEE Trans. Consumer Electron., vol. 39, pp , Aug [13] Y. Wang and O. Lee, Active mesh A feature seeking and tracking image sequence representation scheme, IEEE Trans. Image Processing, vol. 3, pp , Sept [14] W. B. Thompson, K. M. Mutch, and V. A. Berzins, Dynamic occlusion analysis in optical flow fields, IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-7, pp , [15] M. Hoetter and R. Thoma, Image segmentation based on objectoriented mapping parameter estimation, Signal Process., vol. 15, pp , [16] M. Irani, B. Rousso, and S. Peleg, Computing occluding and transparent motions, Int. J. Comput. Vis., vol. 12, pp. 5 16, [17] W.-F. Lee and C.-K. Chan, Two-dimensional split and merge algorithm for image coding, Proc. SPIE Conf. Visual Communications and Image Processing 95, May 1995, vol. 2501, pp [18] P. Gerken, Object-based analysis-synthesis coding of image sequences at very low bit-rates, IEEE Trans. Circuits Syst. Video. Technol., vol. 4, pp , June [19] D. T. Lee and B. J. Schachter, Two algorithms for constructing a Delauney triangulation, Int. J. Comput. Inform. Sci., vol. 9, no. 3, [20] B. D. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, in Proc. DARPA Image Understanding Workshop, 1981, pp [21] B. K. P. Horn and B. G. Schunck, Determining optical flow, Artif. Intell., vol. 17, pp , [22] A. M. Tekalp, Digital Video Processing. Englewood Cliffs, NJ: Prentice-Hall, [23] C. Toklu, A. T. Erdem, M. I. Sezan, and A. M. Tekalp, Tracking motion and intensity-variations using hierarchical 2-D mesh modeling for synthetic object transfiguration, Graphic Models Image Process., vol. 58, pp , Nov [24] Y. Altunbasak and A. M. Tekalp, Closed-form connectivity-preserving solutions for motion compensation using 2-D meshes, this issue, pp [25] ISO/IEC JTC1/SC29/WG11 MPEG Document N999, Yucel Altunbasak, for a photograph and biography, see this issue, p A. Murat Tekalp (S 80 M 82 SM 91), for a photograph and biography, see this issue, p