A Fast Block Matching Algorithm Based on the Winner-Update Strategy

Transcription

1 In Proceedings of the Fourth Asian Conference on Computer Vision, Taipei, Taiwan, Jan. 000, Voume, pages A Fast Bock Matching Agorithm Based on the Winner-Update Strategy Yong-Sheng Chenyz Yi-Ping Hungyz Chiou-Shann Fuhz yinstitute of Information Science, Academia Sinica, Taipei, Taiwan zdepartment of Computer Science and Information Engineering, Nationa Taiwan University, Taipei, Taiwan ABSTRACT Bock matching is a popuar and powerfu technique for stereo vision, visua tracking, object recognition, and video compression. This paper presents a new fast agorithm, which is caed the winner-update agorithm, for bock matching. We utiize an ascending ist of the ower bounds of the matching error for each search position. The cacuation of the matching error can be avoided if one of its ower bound is arger than the gobay minimum matching error. The winner-update agorithm computes the ower bounds ony when the previous ower bounds in the same ist are smaer than the gobay minimum matching error. This agorithm can significanty speed up the computation of the bock matching because (1) the computationa cost of each ower bound is ess than that of the matching error; and () for many search positions, ony the first severa ower bounds in the ist need to be cacuated. In the appication to motion vector estimation, our experiments on test image sequences show that up to 98% of operations can be reduced whie sti guaranteeing the minimum matching error. bock matching, fast agorithm, motion esti- Keywords: mation. 1. INTRODUCTION Bock matching is a popuar and powerfu technique for stereo vision, visua tracking, object recognition, and video compression. In this paper, we focus on its appication to motion vector estimation for video compression. Motioncompensated predictive coding has been widey used in the transmission and storage of video data [5, 6, 3, 4] for reducing the tempora redundancy. For this purpose, the image is usuay divided into bocks and the dispacement (motion vector) of each bock from its corresponding bock in the reference image is estimated and coded. The residua matching error between these two corresponding bocks are aso coded in the data stream to retain the video quaity. Bock matching technique is much used for estimating the bock motion vector due to its simpicity. The matching error between the bock at position (x y) in the current image, I t, and the candidate bock at position (x + u y + v) in the reference image, I t;1, is often defined as the sum of absoute difference (SAD): XX SAD (x y) (u v) = B;1 B;1 i=0 j=0 ji t(x+i y +j);i t;1(x+u+i y +v +j)j (1) where B is the bock size. The best estimate of the bock motion vector, (^u ^v), ocates the bock at position (x+^u y+ ^v) having the minimum matching error, SAD (x y) (^u ^v). This motion vector (^u ^v) can be obtained by using the fusearch (FS) agorithm to cacuate and compare the matching error for each search position in the reference image: (^u ^v) = arg min SAD (x y) (u v) (u v) where (u v) f(u v)j ;R u v Rg, R is the search range, and (x + u y + v) is a vaid position in the reference image, I t;1. This straightforward method takes extremey arge amount of computation, athough it can find the best matching bock. In the iterature, many techniques have been deveoped to reduce the computationa cost of bock matching. These techniques can be cassified into three categories. The techniques in the first category concern the search strategy, that is, they save the computations by reducing the number of positions searched. The gradient descent techniques, based on the unimoda error surface assumption, beong to this category. We-known exampes incude the foowing: the three-step search (TSS) agorithm [8], the two-dimensiona ogarithmic search agorithm [7], and the conjugate direction search agorithm [15]. For another exampe, the search region can aso be restricted to a sma region ocated by the motion vectors of the spatiay and temporay adjacent bocks together with the hierarchica reated bocks [1]. The techniques in the second category, on the other hand, speed up the cacuation of matching error for each search position. One simpe way is to subsampe the pixes in the matching bocks [1], thus ony a part of the pixes are used for cacuating the matching error. Another exampe is the so-caed eary jump-out technique [], which can interrupt the process of matching error accumuation when the accumuated matching error is arge enough comparing to a threshod sequence. The techniques in the first and second categories are not excusive and they can be combined to further improve the efficiency as described in [1, 14]. For another exampe, the hierarchica method [13] estimates the coarse resut of the motion vector in the ower resoution image. Then the resut is refined in the higher resoution image within a sma search 1

2 region centering at the coarse resut. The major drawback of the techniques of the first two categories is that they cannot guarantee to achieve the minimum matching error because ony a part of information is examined. Consequenty, the best coding quaity cannot be obtained. The techniques in the third category use some matching criteria to eiminate search positions whie sti ensuring that the minimum matching error can be obtained [10, 11, 9]. These matching criteria are induced from Minkowski s inequaity and their measures are smaer than or equa to the matching error. At first, the minimum matching error is initiaized as the matching error cacuated at the predicted position. For each search position other than the predicted one, the cacuation of the matching error can be avoided if one of the measures of the matching criteria is arger than the up-to-date minimum matching error. Otherwise, the matching error is cacuated and the up-to-date minimum matching error is updated if necessary. Because cacuating the measure of the matching criteria costs ess than cacuating the matching error, the tota amount of computation can be reduced. When the motion vector is hard to predict, for exampe, if the image sequences contains objects with abrupt motion, the initia minimum matching error may be arge. Hence much useess cacuation of the matching criteria and the matching errors wi be performed unti a position with sma matching error is examined. In the worst case, the matching errors are monotonicay decreasing in the order that the search positions are examined. A the matching criteria are useess and the cacuation of their measures is ony waste of time. In such an unfortunate case, the computationa cost wi be higher than that of using the FS agorithm in fact. In this paper, we propose an efficient and simpe agorithm, named the winner-update agorithm, which can acceerate the computation of the bock matching whie sti ensuring that the minimum matching error can be obtained. For each search position, an ascending ist of the ower bounds of the matching error can be estabished. The cacuation of the matching error can be avoided if one of the ower bounds is arger than the gobay minimum matching error. The ower bound of the matching error can be derived from partia accumuation or Minkowski s inequaity and its computationa cost is ess than that of the matching error. By using the proposed agorithm, very few ower bounds and matching errors are actuay cacuated. Consequenty, the tota computationa cost can be significanty reduced. The agorithm does not need the prediction of the motion vector and it can avoid the useess cacuation of the matching error which the techniques in the third category may suffer. Moreover, this agorithm can be easiy combined with the techniques in the first two categories for further speedup, but the guarantee of the minimum matching error is given up, of course. As an exampe, the combination of the three-step search agorithm and the proposed agorithm is aso presented in this paper.. THE WINNER-UPDATE ALGORITHM Concept In this section, we use a simpe game of poker cards to iustrate the concept of the winner-update agorithm. Suppose there are r payers in the game and each payer is deat d cards. An exampe is shown in Figure 1 where r =5and d =4. The vaue of each card ranges from 1 to 13. The penaty score of each payer is the sum of the vaues of his/her hand and the payer with the minimum penaty score is the winner. The basic idea is that one does not have to cacuate the summation of a the card vaues for each payer before he can determine the winner. If the intermediate summation, or the ower bound of the tota penaty score, for one payer has aready been greater than the tota penaty score of the winner, then he/she has no chance to win and hence we can stop cacuating his/her penaty score to save some computation. Of course, the penaty score of the winner is not known in advance. But the winner-update strategy expained beow can hep to sove the probem. card #4 card #3 card # card # P1 P P 3 P 4 P 5 Figure 1. A simpe game for iustrating the concept of the winner-update agorithm. There are ve payers, P 1,..., P 5, and each payer is deat four cards. Payer 3 has nished the accumuation of the vaues in his/her hand and becomes the winner. The temporary accumuations of others are a greater than the third payer's na accumuation resut. Hence their remaining cacuation can be saved. At the beginning, we ay a the cards face down except the first one of each hand. The ower bound of the penaty score for each payer is initiaized as the vaue of the first card. Ony the temporary winner among the payers having the minimum ower bound is aowed to turn up the face of the next card of his/her hand and update (increase) his/her intermediate ower bound of the penaty score. A new temporary winner is then seected and this process is repeated unti the temporary winner has no card aid facedown and becomes the fina winner. Tabe 1 ists the operations step by step to demonstrate the process for the exampe given in Figure 1. In bock matching for motion vector estimation, the matching errors between the tempate bock and most of the candidate matching bocks are usuay very arge compared with the minimum matching error. That is, most of the payers hod cards with reativey arge vaues except the winner and a few tough competitors. Therefore, by us-

3 operation/status P 1 P P 3 P 4 P 5 initiaization turn up card # of P turn up card # of P turn up card #3 of P turn up card # of P turn up card # of P turn up card #4 of P turn up card #3 of P P 3 is the winner Tabe 1. Step by step cacuation of the penaty score of each payer. The score numbers in bodface are the temporary minima after each step. ing this winner-update strategy, the amount of the cards aid facedown which represents the saved computations can be enormous. Partia Accumuation The process of choosing the winner having minimum penaty score in the previousy mentioned game resembes the process of finding the corresponding bock having minimum matching error in bock matching. Each payer in the game stands for a search position and the penaty score is the difference between the vaues of the corresponding pixes. From Eq.(1), we can define the partia accumuation of pixes as: X PSAD (x y)(u v) = m=0 ji t(x + i m y+ j m) ; I t;1(x + u + i m y+ v + j m)j where f(i m j m)jm = 0 ::: B ; 1g is the index set of a the pixes in the bock. Obviousy, the foowing inequaity reationship hods true (subscript (x y) is dropped for simpicity): PSAD 0 (u v) PSAD 1 (u v) PSAD B ;1 (u v) =SAD(u v): As a resut, the partia accumuation of pixes, PSAD (u v), can be defined as the th eement in the ower bound ist, LB (u v). Notice that the ast eement in the ist, LB B ;1 (u v), equas the matching error, SAD(u v). These B ; 1 ower bounds are in ascending order: LB 0 (u v) LB 1 (u v) LB B ;1 (u v): Furthermore, the computationa cost of these ower bounds are ascending from 1 to B. Genera Winner-Update Agorithm For each bock at position (x y) in the current image, the ower bound of the matching error, LB(u v), between this bock and the candidate matching bock at position (x+u y+ v) in the reference image is initiaized as the first eement in the ower bound ist, LB 0 (u v). An additiona variabe, (u v), is used to record the index in the ist that LB(u v) is assigned and it is initiaized to 0. At first, the search position (^u ^v) is chosen to be the temporary winner having minimum LB(^u ^v). Then the temporary winner updates its ower bound LB(^u ^v) with the next eement in the ower bound ist, LB (^u ^v)+1 (^u ^v), and records the new index number. The new search position (^u ^v) having minimum LB(^u ^v) is chosen again as the new temporary winner. This process is repeated unti (^u ^v) of the new winner equas K, which denotes the number of the ast eement in the ower bound ist. Remember that the ast ower bound, LB K (^u ^v), equas the matching error SAD(^u ^v) at position (^u ^v). The bock matching process can now stop because the ower bounds of the matching errors for other search positions are a arger than the matching error of the winner. The proposed agorithm is given beow: The Winner-Update Agorithm given tempate bock at position (x y) in I t begin for each (u v) in the search range do begin (initiaization) cacuate LB 0 (u v) LB(u v) :=LB 0 (u v) (u v) :=0 end seect (^u ^v) having minimum LB(^u ^v) to be the temporary winner whie (^u ^v) <Kdo begin (^u ^v) :=(^u ^v)+1 cacuate LB (^u ^v) (^u ^v) LB(^u ^v) :=LB (^u ^v) (^u ^v) seect (^u ^v) having minimum LB(^u ^v) to be the new temporary winner end output (^u ^v) end Minkowski s Inequaity Recenty, Lee and Chen [9] deveoped the bock sum pyramid (BSP) agorithm for fast motion vector estimation. They constructed a pyramid structure for each matching bock and defined a matching criterion for each ayer in the pyramid as mentioned in Section 1. This kind of matching criteria can be used as the ower bound we need and is presented beow. Assume the size of the matching bock is K K,we construct K +1ayers of images, each ayer is of furesoution, for each image in the sequence in the foowing way. Consider a four-ayer exampe as iustrated in Figure. For each pixe, I t (x y), at ayer, its vaue is assigned to be the sum of its four corresponding pixes at ayer +1: I t (x y) =It +1 (x y)+i t +1 (x + K;;1 y)+ I t +1 (x y + K;;1 )+It +1 (x + K;;1 y+ K;;1 ) () where is the ayer number, =0 ::: K ; 1, and I t K is the origina image at time t. Notice that the vaue of the pixe I t (x y) at ayer is in fact the sum of the corresponding 3

4 K; K; pixes of the origina image I t K (x y). In this way, we can construct the bock sum pyramids for a the positions in the whoe image, except for the ast K; ; 1 pixes at each row and each coumn due to the boundary condition. Figure 3 shows an exampe of 4-ayer images x x+1 x+ x+3 x+4 x+5 x+6 x+7 x+8... Figure. This gure iustrates the 4-ayer images constructed from an origina image, here each ayer is of fu-resoution. Ony one dimension (the x-axis) is shown for simpicity. Two pyramids at position x and x +1 are depicted in soid ine and dotted ine, respectivey. I 0 I 1 The number of pixes used for cacuating SAD (x y) (u v) at ayer is, that is, the number is 1 4 ::: K K when is 0 1 ::: K, respectivey. We use SAD (x y) (u v) defined in Eq.(3) as the ower bound LB (u v) of the matching error SAD (x y) (u v) at position (x y). 3. COMBINATION WITH THE THREE-STEP SEARCH ALGORITHM The proposed winner-update agorithm can be easiy combined with other fast agorithms for further speedup, but the guarantee of the minimum matching error is given up. Considering the we-known TSS agorithm, nine search positions which are coarsey spaced are examined at first. Then the position having minimum matching error is seected and eight search positions which are ess coarsey spaced around this seected position are examined. The matching errors of these eight positions and the previousy seected position are compared and a new position having minimum matching error is seected again. This process is repeated unti the required resoution (spacing) of the examined search positions is achieved. At each iteration, the TSS agorithm needs to cacuate and compare the matching errors of nine or eight search positions. Meanwhie the winner-update agorithm can be appied to efficienty find the one having minimum matching error among these search positions. The accuracy of this combined agorithm is the same as that of the TSS agorithm because the winner-update agorithm can find the best matching position at each iteration. 4. EXPERIMENTS Impementation Issues I I 3 Figure 3. This gure presents the construction of 4-ayer images. At ayer, the matching error SAD (x y) (u v) between the bock at position (x y) in image I t and the candidate matching bock at position (x + u y + v) in image I t;1 is defined as: ;1 SAD (x y) (u v) = X ;1 X i=0 j=0 ji t (x+ K; i y + K; j); I t;1 (x + u + K; i y + v + K; j)j (3) where =0 ::: K. According to Minkowski s inequaity (1-norm case here), Lee and Chen derived and proved the foowing inequaity reationship between the matching errors computed at each ayer: SAD (x y) 0 (u v) SAD (x y) 1 (u v) SAD (x y) K (u v): (4) The major overhead of the winner-update agorithm is the seection of the minimum ower bound at each iteration. A heap data structure can be used in this agorithm and O(og n) operations are used to maintain the heap condition and keep the eement with the smaest ower bound in the root of the heap tree at each iteration. The seection overhead can be reduced to constant time by using hashing method. To reduce the size of hashing tabe, the mean absoute difference (MAD) is used instead of SAD and the ower bound can be cacuated as (subscript (x y) is dropped for simpicity): LB (u v) = MAD (u v) = 1 K K SAD (u v): Because MAD (u v) ranges from 0:0 to 55:0, a tabe with 56 eements is used and the tabe address is assigned with the integer part of MAD (u v). Separate chaining is used in our impementation to resove the coision. The first bock in the first non-empty tabe entry is chosen to be the temporary winner. After updating this bock s new ower bound, it is moved, if necessary, to the corresponding tabe entry according to its new ower bound in constant time. When the seected bock with minimum ower bound reaches 4

5 ayer 0, a the bocks with the same tabe address, that is, in the same chain, are examined to determine the fina winner with the minimum matching error. Experimenta Resuts This section shows the experimenta resuts of appying the winner-update agorithm to some test image sequences. Five agorithms the fu-search (FS) agorithm, the bock sum pyramid (BSP) agorithm [9], the proposed winner-update agorithm with the ower bound defined from Minkowski s inequaity (WinUpMI), the three-step search (TSS) agorithm [8], and the combination of the winner-update agorithm and the three-step search agorithm (WinUpTSS) were impemented. Notice that the ast two agorithms do not guarantee that the goba minimum can be found. We evauate these agorithms by comparing three performance issues: (1) the peak signa-to-noise ratio (PSNR) between the reconstructed motion-compensated image and the origina image; () the number of absoute operations for cacuating the matching error; and (3) the execution time according to our impementation and hardware environment. These agorithms were impemented in C anguage on a Sun Utra-1 workstation. The execution time incudes reading images, muti-ayer images construction if necessary, and motion vector estimation. As shown in Figure 4, five image sequences Saesman, Trevor, Coastguard, Footba, and Foreman were used for comparing the performance of the agorithms. Each image was divided into nonoverapping bocks and the search range was set to [;16 16] [;16 16]. in this sequence contains compex background as shown in Figure 4(a) and the PSNR vaue is high. Thus ony a few competitors with sma matching error exist and the minimum matching error is sma. Most search positions obtained arger ower bounds than the sma minimum matching error and withdrew from the competition. Ony 1.97% of absoute operations are needed to cacuate the matching errors, comparing to the absoute operations for the FS agorithm. Considering the overhead of constructing the muti-ayer images and switching the cacuation among different search positions, the tota execution time for estimating the motion vectors with the winner-update agorithm remain to be very sma. Our experiments shows that the WinUpMI agorithm costs ony 3.97% execution time of what the FS agorithm costs. As for the WinUpTSS agorithm, further speedup can be achieved without decreasing the PSNR vaue of the TSS agorithm. The number of absoute operations decreases from 3.11% to 0.63% and the actua execution time decreases from 3.33% to 1.61%. Agorithm (db) number % sec. % FS BSP WinUpMI TSS WinUpTSS Tabe. Performance comparison of ve agorithms with Saesman image sequence. (a) (c) (b) (d) Tabe 3 shows the experimenta resuts obtained with the Trevor image sequence which contains 99 images of size As shown in Figure 4(b), the background of the images contains periodic strips. The bocks in the background area needed more computation to find the fina corresponding bock having the minimum matching error. Thus higher operation ratio and execution time ratio were obtained comparing to those for the Saesman image sequence. Agorithm (db) number % sec. % FS BSP WinUpMI TSS WinUpTSS Tabe 3. Performance comparison of ve agorithms with Trevor image sequence. (e) Figure 4. Five test image sequences: (a) Saesman, (b) Trevor, (c) Footba, (d) Foreman, and (e) Coastguard. Tabe shows the performance comparison of the abovementioned agorithms with the Saesman image sequence which contains 100 images of size The image The experiments described beow use three image sequences containing arger motion. The first image sequence Coastguard contains 300 images of size , the second image sequence Footba contains 60 images of size 35 40, and the third image sequence Foreman contains 400 images of size Tabes 4, 5, and 6 show the experimenta resuts using the three image sequences. Because of the ower PSNR vaue, the efficiency improvement of the WinUpMI agorithm comparing to the FS agorithm is not as significant as that in the experiments of Saesman 5

6 and Trevor image sequences. The minimum matching errors are arger and more computations are required to determine the best matching bocks. It is obvious that the proposed WinUpMI agorithm outperforms the BSP agorithm in these experiments especiay when there is a ot of arge motion in the image sequence. Agorithm (db) number % sec. % FS BSP WinUpMI TSS WinUpTSS Tabe 4. Performance comparison of ve agorithms with Coastguard image sequence. Agorithm (db) number % sec. % FS BSP WinUpMI TSS WinUpTSS Tabe 5. Performance comparison of ve agorithms with Footba image sequence. Agorithm (db) number % sec. % FS BSP WinUpMI TSS WinUpTSS Tabe 6. Performance comparison of ve agorithms with Foreman image sequence. 5. CONCLUSIONS In this paper, we have proposed a new fast agorithm, named the winner-update agorithm, which can speed up the computation of bock matching whie sti guaranteeing the minimum matching error. According to our experiments, the proposed WinUpMI agorithm can save 91.7% to 98% of absoute operations, depending on the image sequence. If the goba optimum is not required, then WinUpTSS can save 98.6% to 99.4% of absoute operations without decreasing the PSNR vaue of using TSS agorithm. The WinUpMI agorithm requires a preprocessing stage to construct muti-ayer images. Additiona operations are aso required to switch among the search positions. After considering these two kinds of overhead, the efficiency of the proposed agorithm is sti very good. According to our experiments, 86.7% to 96% of the execution time can be saved, compared to that using the FS agorithm. References [1] J. Chaidabhongse and C.-C. J. Kuo. Fast motion vector estimation using mutiresoution-spatio-tempora correations. IEEE Transactions on Circuits and Systems for Video Technoogy, 7(3): , [] H.-C. Huang and Y.-P. Hung. Adaptive eary jump-out technique for fast motion estimation in video coding. Graphica Modes and Image Processing, 59(6): , [3] ISO/IEC Information technoogy coding of moving pictures and associated audio for digita storage media at up to about 1.5 mbit/s - part : Video [4] ISO/IEC and ITU-T Recommendation H.6. Information technoogy generic coding of moving pictures and associated audio information: Video. [5] ITU-T Recommendation H.61. Video codec for audiovisua services at p64 kbit/s. Mar [6] ITU-T Recommendation H.63. Video coding for ow bit rate communication. Feb [7] J. R. Jain and A. K. Jain. Dispacement measurement and its appication in interframe image coding. IEEE Transactions on Communications, COM-9(1): , [8] T. Koga, K. Iinuma, A. Hirano, Y. Iijima, and T. Ishiguro. Motion-compensated interframe coding for video conferencing. In Proceedings of Nationa Teecommunications Conference, voume 4, pages G5.3.1 G5.3.5, New York, [9] C.-H. Lee and L.-H. Chen. A fast motion estimation agorithm based on the bock sum pyramid. IEEE Transactions on Image Processing, 6(11): , [10] W. Li and E. Saari. Successive eimination agorithm for motion estimation. IEEE Transactions on Image Processing, 4(1): , [11] Y.-C. Lin and S.-C. Tai. Fast fu-search bock-matching agorithm for motion-compensated video compression. IEEE Transactions on Communications, 45(5):57 531, [1] B. Liu and A. Zaccarin. New fast agorithms for the estimation of bock motion vectors. IEEE Transactions on Circuits and Systems for Video Technoogy, 3(): , [13] K. M. Nam, J.-S. Kim, R.-H. Park, and Y. S. Shim. A fast hierarchica motion vector estimation agorithm using mean pyramid. IEEE Transactions on Circuits and Systems for Video Technoogy, 5(4): , [14] Y. Q. Shi and X. Xia. A threshoding mutiresoution bock matching agorithm. IEEE Transactions on Circuits and Systems for Video Technoogy, 7(): , [15] R. Srinivasan and K. R. Rao. Predictive coding based on efficient motion estimation. IEEE Transactions on Communications, COM-33(8): ,