Robust Motion Estimation for Video Sequences Based on Phase-Only Correlation

Size: px

Start display at page:

Download "Robust Motion Estimation for Video Sequences Based on Phase-Only Correlation"

Stephanie Walsh
6 years ago
Views:

Robust Motion Estimation for Video Sequences Based on Phase-Only Correlation Loy Hui Chien and Takafumi Aoki Graduate School of Information Sciences Tohoku University Aoba-yama 5, Sendai, 98-8579,

1 Robust Motion Estimation for Video Sequences Based on Phase-Only Correlation Loy Hui Chien and Takafumi Aoki Graduate School of Information Sciences Tohoku University Aoba-yama 5, Sendai, , Jaan ABSTRACT In this aer, we resent a robust Phase-Only Correlation (POC)-based motion estimation method for video sequences. Robust motion estimation is indisensable for many alications such as mesh-based video coding, stereo vision, and suer-resolution imaging. In our roosed method, the motion vector of a oint in a video frame is adatively switched between motion vectors obtained by two motion estimation methods: (i) POC-based full search and (ii) POC-based hierarchical search. This aroach can reliably detect both global and local motion in video sequences. We evaluate the robustness of our roosed method in mesh-based motion comensation. Exerimental results show that our roosed method erforms significantly better than conventional full search using Sum of Absolute Differences (SAD). KEY WORDS robust motion estimation, hase-only correlation, SAD, full search, hierarchical search Introduction Motion estimation is the rocess of determining the movement of the objects of a video sequence. The movement is usually exressed in terms of the motion vectors of selected oints within the current frame with resect to another frame known as the reference frame. A motion vector reresents the dislacement of a oint between the current frame and the reference frame. Motion estimation is a fundamental task in numerous fields, such as image rocessing, image analysis, video coding, and comuter vision. Robust highaccuracy motion estimation is essential for alications such as mesh-based motion comensation for video coding [], stereo vision 3D measurement [2], and suer-resolution imaging [3] (the reconstruction of a high-resolution image using multile low-resolution images). Here, robustness refers to consistent ixellevel estimation of motion vectors with minimal false detection. Among the various motion estimation methods, the block matching algorithm is most oular due to its simlicity. In block matching algorithms, an image block centered on a oint in the current frame is comared with candidate blocks in the reference frame based on certain dissimilarity or similarity measures in order to find the best matching block within a redefined search area. An examle of a dissimilarity measure is Sum of Absolute Differences (SAD). The motion vector of the oint is given by the block dislacement. Strategies for finding the best matching block are broadly classified into two tyes: full search methods and hierarchical search methods. The former is suitable for detecting local motion of individual objects, while the latter is suitable for detecting global motion of the scene. Recently, a high-accuracy image matching technique using a Phase-Only Correlation (POC) function [4] [6] has been develoed. Using the POC function, we can estimate the translational dislacement as well as the degree of similarity between two image blocks from the location and height of the correlation eak, resectively. It has been demonstrated that this matching technique can estimate the dislacement between two images with /-ixel accuracy when the image size is about ixels [5]. This aer resents a robust POC-based motion estimation method for video sequences. Our roosed method combines the advantages of hierarchical search with those of full search. The motion vector of a oint is adatively switched between motion vectors obtained from POC-based hierarchical search and POC-based full search, so as to detect both global and local motion in video sequences. We evaluate the robustness of our roosed method using mesh-based motion comensation, where the quality of the motion comensated images is highly deendent on the robustness of the motion estimation method. 2 Phase-Only Correlation Consider two N N 2 images, f(n,n 2 )andg(n,n 2 ), where we assume that the index ranges are n = M,,M and n 2 = M 2,,M 2 for mathematical simlicity, and hence N =2M + and N 2 =

2 2M 2 +. Let F (k,k 2 )andg(k,k 2 ) denote the 2D Discrete Fourier Transforms (2D DFTs) of the two images. F (k,k 2 )andg(k,k 2 )aregivenby F (k,k 2 ) = f(n,n 2 )W kn N W k2n2 N 2 n n 2 = A F (k,k 2 )e jθf (k,k2), () G(k,k 2 ) = g(n,n 2 )W kn N W k2n2 N 2 n n 2 = A G (k,k 2 )e jθg(k,k2), (2) where k = M,,M, k 2 = M 2,,M 2, 2π 2π j N j W N = e N, W N2 = e 2, and the oerator n n 2 denotes M M2 n = M n 2= M 2. A F (k,k 2 )and jθf (k,k2) A G (k,k 2 ) are amlitude comonents, and e and e jθg(k,k2) are hase comonents. The cross-hase sectrum (or normalized cross sectrum) ˆR(k,k 2 ) is defined as ˆR(k,k 2 ) = F (k,k 2 )G(k,k 2 ) F (k,k 2 )G(k,k 2 ) = e jθ(k,k2), (3) where G(k,k 2 ) denotes the comlex conjugate of G(k,k 2 )andθ(k,k 2 )=θ F (k,k 2 ) θ G (k,k 2 ). The Phase-Only Correlation (POC) function ˆr(n,n 2 )is the 2D Inverse Discrete Fourier Transform (2D IDFT) of ˆR(k,k 2 )andisgivenby ˆr(n,n 2 ) = N N 2 k k 2 ˆR(k,k 2 )W kn N W k2n2 N 2, (4) where k k 2 denotes M M2 k = M Now consider f c (x,x 2 ) as a 2D image defined in continuous sace with real-number indices x and x 2.Letδ and δ 2 reresent sub-ixel dislacements of f c (x,x 2 )tox and x 2 directions, resectively. So, the dislaced image can be reresented as f c (x δ,x 2 δ 2 ). Assume that f(n,n 2 )andg(n,n 2 ) are satially samled images of f c (x,x 2 )andf c (x δ,x 2 δ 2 ), and are defined as k 2= M 2. f(n,n 2 ) = f c (x,x 2 ) x=n T,x 2=n 2T 2, g(n,n 2 ) = f c (x δ,x 2 δ 2 ) x=n T,x 2=n 2T 2, where T and T 2 are the satial samling intervals, and index ranges are given by n = M,,M and n 2 = M 2,,M 2. The POC function ˆr(n,n 2 ) between f(n,n 2 )andg(n,n 2 ) will be given by ˆr(n,n 2 ) α sin{π(n + δ )} sin{π(n 2 + δ 2 )} N N 2 sin{ π N (n + δ )} sin{ π N 2 (n 2 + δ 2 )},(5) where α. The eak osition of the POC function corresonds to the dislacement between the two images, and the eak value α corresonds to the degree of correlation between the two images. ( n,n 2 ) ^r n n Figure. Function fitting for estimating the eak osition. For high accuracy sub-ixel image matching, the following techniques [5] are imortant: function fitting using equation (5) for high-accuracy estimation of subixel dislacement (δ,δ 2 )andeakvalueα, alication of Hanning window to reduce image boundary effects and low-ass filtering to reduce aliasing and noise effects. Figure shows an examle of function fitting using equation (5) to estimate the true osition and height of the correlation eak. In this aer, we use two versions of the POC function for image matching. We use a simlified version where only windowing technique and low-ass filtering is emloyed for fast comutation. The dislacement between two image blocks is estimated by detecting the osition of the maximum value of the POC function ˆr(n,n 2 ) in equation (4) with ixel accuracy. We also emloy a high-accuracy version that uses function fitting, windowing and low-ass filtering technique for high-accuracy estimation of correlation eak α and sub-ixel dislacement (δ,δ 2 ). 3 POC-Based Motion Estimation In this section, our roosed POC-based motion estimation method is described. We describe two different search strategies for block matching: POC-based full search and POC-based hierarchical search. We also roose an adative search strategy that adatively switches between the result of the above two algorithms for more robust motion estimation. 3. Full Search Method In this section, we describe a POC-based full search motion estimation method (known thereafter as POC- FS) that requires less calculation than conventional full search while having almost no loss in otimality

3 We consider two image blocks of size W W with Hanning window function of the same size alied. Since the half-width of the Hanning window function is W 2, we may consider the maximum reliable dislacement estimate between the two image blocks to be ± W 4 both horizontally and vertically. Hence, instead of examining every candidate block, we only need to examine candidate blocks at W 4 ixel intervals in the search area. Procedure for POC-FS Inut: current image I(n,n 2 ), reference image J(n,n 2 ), oint in I(n,n 2 ) Outut: corresonding oint q of oint in J(n,n 2 ), motion vector v FS of oint Ste : Extract an image block of size W W centered on oint in the current frame I(n,n 2 ). Ste 2: Calculate POC function between the image block in the current frame I(n,n 2 ) and candidate blocks in the reference frame J(n,n 2 )every W 4 ixels in the search area. We use the simlified version of the POC function to obtain the rough correlation eak (defined as the maximum value of the POC function ˆr(n,n 2 ) in Equation (4)) and the eak osition which gives the dislacement between the two blocks with ixel accuracy. Ste 3: Identify the to 3 matching blocks in Ste 2, and dislace the blocks by their dislacement estimates. This brings the correlation eaks to the centers of the blocks. Re-calculate the POC function for the 3 new candidate blocks. The best matching block is the block with the highest correlation eak among the 3 blocks. Again, the simlified version of the POC function is used. The corresonding oint q is centered on the best matching block. Ste 4: Find the motion vector v FS = q. In our exeriments we set the window size as and the search range as ±6 or ±32 both horizontally and vertically, deending on the exected range of motion. 3.2 Hierarchical Search Method In the POC-based hierarchical search motion estimation method (known thereafter as POC-HS), some coarser versions of the original inut images are created. This method is also known as the coarse-to-fine corresondence search technique [6]. The POC-based block matching starts at the coarsest image layer and the oeration gradually moves to the finer layers. The motion vector detected at each layer is roagated to the next finer layer in order to guide the search at that layer. An overview of the technique is shown in Figure 2. Let o be the given oint in the current image, and q be the corresonding oint in the reference image, and let l and q l be the matching oints at the l-th layer. The aim of the corresondence search is to find the corresonding oint q of oint andindoingso, we obtain the motion vector of as q. Procedure for POC-HS Inut: current image I (n,n 2 )(=I(n,n 2 )), reference image J (n,n 2 )(=J(n,n 2 )), oint (= ) ini (n,n 2 ) Outut: corresonding oint q of oint in J (n,n 2 ), motion vector v HS of oint Ste : For l =, 2,,l max,createthel-th layer images I l (n,n 2 )andj l (n,n 2 ), i.e., coarser versions of I (n,n 2 )andj (n,n 2 ), recursively as follows: I l (n,n 2 ) = 4 J l (n,n 2 ) = 4 i = i 2= i = i 2= I l (2n + i, 2n 2 + i 2 ), J l (2n + i, 2n 2 + i 2 ). In our exeriments we set the value of l max to2or3 deending on the exected range of motion. Ste 2: For every layer l =, 2,,l max,calculate the coordinate l =( l, l2 ) corresonding to the original oint recursively as follows: l = 2 l =( 2 l, 2 l 2 ). (6) Ste 3: We assume that q lmax = lmax in the coarsest layer. Let l = l max. Ste 4: From the l-th layer images I l (n,n 2 ) and J l (n,n 2 ), extract two image blocks (of size W W ) f l (n,n 2 )andg l (n,n 2 ) with their centers on l and 2q l+, resectively. For accurate matching, the size of image blocks should be reasonably large. In our exeriments, we use image blocks. Ste 5: Estimate the dislacement between f l (n,n 2 ) and g l (n,n 2 ) with ixel accuracy using the simlified l max l+ l l- l l l+ q l+ To layer l- I l+ J l+ I l f l (n,n 2 ) POC 2q l+ g l (n,n 2 ) q l = 2q l+ + δ l J l To layer l- Figure 2. Block matching using a hierarchical coarseto-fine aroach. 443

4 version of the POC function, in which the dislacement is determined to ixel-level accuracy. Let the estimated dislacement vector be δ l. The l-th layer corresondence q l is determined as follows: q l = 2q l+ + δ l. (7) Ste 6: Decrement the counter by as l = l and reeat from Ste 4 to Ste 6 while l. Ste 7: Find the motion vector v HS = q. 3.3 Adative Search Method To adat to both global and local motion in video sequences, we roose a POC-based adative search motion estimation method (known thereafter as POC- HS/FS) that adatively switches between POC-HS and POC-FS to detect correct motion. When switching between POC-HS and POC-FS, two factors come into consideration. The first factor is how well the image block matches with its best matching block. This is given by the correlation eak α of the POC function. The second factor is to what extent the motion vector at a oint correlates with the motion vectors of the set of its surrounding oints S. We introduce a measure D, which reresents how much the motion vector v differs from the motion vectors of the oints in S. S may not necessarily refer to the adjacent ixels of. We can freely define the surrounding oints to have some distance from. D(v )= s S v v s. (8) POC-FS is effective for detecting local motions of individual objects, but the ossibility of mismatching with a similar-looking block is high. On the other hand, POC-HS uses a larger matching area to increase the reliability, but may fail to detect the correct motion when the block overlas with two or more objects with different motion. We use the following discriminant Z for each oint to decide which result to use. reference image J(n,n 2 ), oint in I(n,n 2 ) Outut: motion vector v of oint Ste : Use POC-HS to find v HS and α HS. Ste 2: Use the following rocedure to obtain v. If α HS v = v HS else > threshold value k then Use POC-FS to find v FS Calculate discriminant Z If Z. then v = v FS else and α FS v = v HS end end In our exeriments we let k =.5. An examle of motion estimation using POC-FS, POC-HS and POC-HS/FS is shown in Figure 3 for the video sequence F lower Garden. POC-FS fails to detect the correct motion for some oints close to the tree boundary because of occlusion. On the other hand, due to different motion of the tree and the background, POC-HS fails to detect the correct motion for some oints within the tree. By adatively switching between POC-FS and POC-HS, a more satisfactory result is obtained by POC-HS/FS. 4 Exeriment and Evaluation In our first exeriment we investigate the robustness of POC-HS/FS for mesh-based motion comensation using a wide variety of MPEG-4 standard test video sequences that have mostly global motion, local motion or a good mix of both. We create the motion comensated image of the current frame and comare it with the current frame to calculate PSNR. The motion comensation rocedure for the current frame t is as follows. Nodes are ositioned every 6 ixels to form a square mesh that artitions the image into square blocks. To avoid boundary occlusion effects, we exclude a frame boundary of 6 ixels Z = αfs α HS ) D(vHS D(v FS ). (9) Here, α FS, α HS are the correlation eaks of the POC function for the best matching block found using POC- FS, POC-HS resectively. We use the function fitting technique described in Section 2 for high-accuracy estimation of the eak values. If Z., then the v FS is chosen, otherwise if Z<., then v HS is chosen. Procedure for POC-HS/FS Inut: current image I(n,n 2 ), (a) (b) (c) Figure 3. Motion estimation for F lower Garden using (a) POC-FS, (b) POC-HS, and (c) POC-HS/FS. 444

5 from the motion comensation rocess. Motion estimation is done for every node in the current frame t with resect to the reference frame t. The rojective transformation of a oint (n,n 2 )inframet to the oint (n,n 2)inframet is defined as n n 2 = h h 2 h 3 h 4 h 5 h 6 n n 2, () h 7 h 8 Reference frame t- (n,n2 ) H H Current frame t (n,n2) Motion comensated image of frame t where h h 8 are the rojective arameters of the rojective transformation matrix H, showninfigure 4. The coordinates of the four corner nodes of an image block in frame t and the coordinates of their corresonding nodes in frame t are used to calculate H. Next, the inverse matrix H is used to create the motion comensated image block of frame t, asshown in Figure 4. This rocedure is reeated for every block to generate the motion comensated image of frame t. To calculate H accurately, sub-ixel accuracy of motion vectors is required. We use the high-accuracy version of the POC function described in Section 2, where function fitting is emloyed to calculate the subixel dislacement after the motion vectors have been estimated to ixel-level accuracy using POC-HS/FS. In our exeriment, we also comare our roosed method with SAD-based full search motion estimation method (known thereafter as SAD-FS). The block size for SAD-FS is set at 6 6, corresonding to the halfwidth of the Hanning window used for POCbased image matching. For sub-ixel estimation of motion vector, we use bilinear interolation to estimate to.25-ixel accuracy. Video sequences often contain uniform, textureless and featureless regions that cause mismatching. An examle of this is the sky in the video sequences Shinjuku and F lower Garden. To revent such mismatching, we identify nodes in areas that have low standard deviation of luminance and set their motion vectors to zero. The average PSNR of the motion comensated images is given in Table. We see that the average PSNR is higher for POC-HS/FS than for SAD-FS. In general, POC-HS/FS roduced more robust estimation of motion vectors than SAD-FS, hence resulting in more accurate motion comensation. For video sequences that contained similar-looking textures such as Shinjuku, Church and Mobile Calendar, POC- HS/FS was significantly more robust, outerforming SAD-FS by 2.5 db. Kiel Harbour, which contains a gradual zoom-in, had only a slightly higher PSNR for POC-HS/FS. In Table, we also observe that the average number of block matches for ixel-level matching is significantly lower for POC-HS/FS. Figures 5 and 6 lot the PSNR er frame for Mobile Calendar and Shinjuku resectively. In both cases the PSNR for POC-HS/FS is consistently higher than for SAD-FS. Figures 7 and 8 give examles of mo- Figure 4. Motion comensation rocedure for a block. tion comensated images. We observe that the motion comensated images for POC-HS/FS are much closer to the original images. In our second exeriment we investigate the effectiveness of the adative switching mechanism of POC- HS/FS. We emloy 5 frames of the video sequence F lower Garden, which contains fast horizontal camera motion, along with an even faster-moving foreground tree object (an examle is showninsection3.3). We call this sequence F lower Garden. We erform the motion comensation rocedure for each current frame t with resect to reference frames t b, whereb =,2,3 and 4, to simulate various degrees of global and local motion. The average PSNR for POC-HS, POC-FS, and POC-HS/FS, along with the ercentage of vectors that were adatively switched from POC-HS to POC-FS are shown in Table 2. We observe that the ercentage of switched vectors increases with larger aarent disarity between the global motion of the background and the local motion of the tree object. For each value of b, the average PSNR for POC-HS/FS is highest, thus demonstrating the effectiveness of the switching mechanism. 5 Conclusion This aer resents a robust POC-based motion estimation method for video sequences. In the roosed method, the motion vector is adatively chosen from the result of POC-based hierarchical search and POCbased full search, so as to accommodate both global and local motion. The roosed method is evaluated using mesh-based motion comensation, where robust motion estimation is essential. We have demonstrated that the roosed method is generally more robust than the conventional SAD-based full search method. Further comarison of POC-based methods with SAD-based methods, as well as the alication of the roosed method to stereo image matching, moving object segmentation, etc. are left for future study. 445

29 28 27 PSNR (db) 26 25 24 23 22 POC HS/FS SAD FS 2 2 4 6 8 Frame Number Figure 5. PSNR of motion comensated images for Mobile Calendar. 36 (a) (b) (c) Figure 7.

PSNR of motion comensated images for Shinjuku. (a) (b) (c) Figure 8. Motion comensation images for Shinjuku: (a) Original image, (b) POC-HS/FS, and (c) SAD-FS.

6 PSNR (db) POC HS/FS SAD FS Frame Number Figure 5. PSNR of motion comensated images for Mobile Calendar. 36 (a) (b) (c) Figure 7. Motion comensation images for Mobile Calendar: (a) Original image, (b) POC-HS/FS, and (c) SAD-FS. PSNR (db) POC HS/FS SAD FS Frame Number Figure 6. PSNR of motion comensated images for Shinjuku. (a) (b) (c) Figure 8. Motion comensation images for Shinjuku: (a) Original image, (b) POC-HS/FS, and (c) SAD-FS. Acknowledgements The authors wish to thank Mr. Satoshi Kondo and the team at AV Core Technology Develoment Center/Image Processing Grou, Matsushita Electric Industrial Co., Ltd., for their kind suort and advice. References [] G. J. Sullivan and R. L. Baker, Motion comensation for video comression using control grid interolation, Proc. IEEE ICASSP, Vol. 4, , May 99. [2] O. D. Faugeras, Three-Dimensional Comuter Vision, MIT Press, 993. [3] C. P. Sung, K. P. Min, and G. K. Moon, Suerresolution image reconstruction: A technical overview, IEEE Signal Processing Magazine, Vol. 2, No. 3,. 2 36, 23. [4] C. D. Kuglin and D. C. Hines, The hase correlation image alignment method, Proc. Int. Conf. on Cybernetics and Society, , 975. [5] K. Takita, T. Aoki, Y. Sasaki, T. Higuchi, and K. Kobayashi, High-accuracy subixel image registration based on hase-only correlation, IEICE Trans. Fundamentals, Vol. E86-A, No. 8, , August 23. [6] K. Takita, M. A. Muquit, T. Aoki, and T. Higuchi, A sub-ixel corresondence search technique for comuter vision alications, IEICE Trans. Fundamentals, August 24,(to be ublished). Table. Average PSNR of motion comensated images for POC-HS/FS, SAD-FS (unit: db). (The average number of block matches for ixel-level matching is shown in arentheses.) Sequence POC-HS/FS SAD-FS Shinjuku (.2 3 ) (2.5 5 ) Church (8.2 2 ) (4. 5 ) Kiel Harbour (.4 3 ) (4. 5 ) F lower Garden (7.8 2 ) (3.3 5 ) Foreman (.5 3 ) (4.6 5 ) Mobile Calendar (. 3 ) (4.8 5 ) Table 2. Average PSNR of motion comensated images for POC-HS, POC-FS, and POC-HS/FS (unit: db) for F lower Garden, and average ercentage of switched vectors. b POC-HS POC-FS POC-HS/FS %

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2 Mingliang Chen 1, Weiyao Lin 1*, Xiaozhen Zheng 2 1 Deartment of Electronic Engineering, Shanghai Jiao Tong University, China