A Fast Sub-pixel Motion Estimation Algorithm for. H.264/AVC Video Coding

Transcription

1 A Fast Sub-pixel Motion Estimation Algorithm or H.64/AVC Video Coding Weiyao Lin Krit Panusopone David M. Baylon Ming-Ting Sun 3 Zhenzhong Chen 4 and Hongxiang Li 5 Institute o Image Communiation and Inormation Proessing Dept. o Eletroni Engineering Shanghai Jiao Tong University Shanghai 4 China Advaned Tehnology Department Mobile Devies & Home Motorola In. San Diego CA 9 USA 3 Dept. o Eletrial Engineering University o Washington Seattle WA 9895 USA 4 Shool o Eletrial and Eletroni Engineering Nanyang Tehnologial University Singapore 5 Dept. o Eletrial and Computer Engineering North Dakota State University Fargo ND USA Abstrat Motion Estimation ME is one o the most time-onsuming parts in video oding. The use o multiple partition sizes in H.64/AVC makes it even more ompliated when ompared to ME in onventional video oding standards. It is important to develop ast and eetive sub-pixel ME algorithms sine a The omputation overhead by sub-pixel ME has beome relatively signiiant while the omplexity o integer-pixel searh has been greatly redued by ast algorithms and b Reduing sub-pixel searh points an greatly save the omputation or sub-pixel interpolation. In this paper a novel ast sub-pixel ME algorithm is proposed whih perorms a rough sub-pixel searh beore the partition seletion and perorms a preise sub-pixel searh or the best partition. By reduing the searhing load or the large number o non-best partitions the omputation omplexity or sub-pixel searh an be greatly dereased. Experimental results show that our method an redue the sub-pixel searh points by more than 5% ompared to existing ast sub-pixel ME methods with negligible quality degradation. I. Introdution H.64/AVC is the state-o-the-art video oding standard established by ITU-T and ISO/IEC.

2 H.64/AVC uses many new tehniques and is able to save more than 5% in bitrate while having similar video quality ompared to the MPEG- video oding standard []. Motion Estimation ME is one o the most time-onsuming parts in video oding. Developing ast algorithms or ME to redue omputational omplexity in video oding has been an important and hallenging problem. In the H.64/AVC Joint Model JM [5] the ME proess ontains two stages: integer pixel searh over a large area and sub-pixel searh around the best seleted integer pixel. Sine H.64/AVC uses 7 partition sizes or inter-rame predition and 4 4 the omplexity o multi-partition ME is high []. It is beoming more ritial to develop ast and eetive sub-pixel ME algorithms or H.64/AVC. Firstly the omputation overhead by sub-pixel ME has beome relatively signiiant while the omplexity o integer-pixel searh has been greatly redued by ast algorithms. For example there have been integer-pixel ME algorithms [4 6] that only need between 3 and 5 integer searh points to alulate the inal integer Motion Vetor MV. The omputation in the 6-point sub-pixel searh method used in the JM thus beomes omparatively large. Seondly typial sub-pixel searhes require interpolating sub-pixel values or omputing the Sum o Absolute Dierene SAD. Reduing sub-pixel searh points an also redue the interpolation omputation time. In this paper a novel sub-pixel ME algorithm is proposed or H.64/AVC whih perorms a rough sub-pixel searh beore the partition seletion and perorms a preise sub-pixel searh or the best partition. By reduing the searhing load or the large number o non-best partitions the omputation omplexity or sub-pixel searh an be greatly dereased. Experimental results show that the proposed algorithm an signiiantly redue the number o sub-pixel searh points ompared to other ast sub-pixel ME algorithms [6-9] with negligible quality degradation. The rest o this paper is organized as ollows. Setion II reviews existing researh on sub-pixel ME. Setion III provides in-depth analysis on how to urther redue the searh points or sub-pixel ME or multiple partitions. The proposed algorithm is desribed in Setion IV. Setion V shows the experimental results and Setion VI onludes the paper.

3 II. Related Work Chen et al. [6] analyzed the dierene between the integer-pixel mathing error surae and the sub-pixel mathing error surae. Aording to Chen s analysis the integer-pixel mathing error surae is ar rom a unimodal surae inside the searhing window due to the omplexity o the video ontent. The assumption o unimodal will easily result in trapping in a loal minimum. However or the sub-pixel mathing error surae the unimodal surae assumption holds in most ases beause o the smaller searh range o sub-pixel ME as well as the high orrelation between sub-pixels due to the sub-pixel interpolation. There has been muh researh on ast sub-pixel ME [6-9 7]. Most o these methods are based on the unimodal surae assumption and perorm the sub-pixel searh in two steps: Predit a sub-pixel MV SPMV and Perorm a small area searh around the SPMV to obtain the inal sub-pixel MV. The method to get the sub-pixel predited MV an be summarized in two ways: using spatiotemporal inormation and modeling the Sum o Absolute Dierene SAD surae. Chen et al. [6] and Yang et al. [8] used spatiotemporal inormation to get the SPMVs. In [6] a Center Biased Frational Pixel Searh CBFPS ast sub-pixel ME method is studied where the MVs o neighboring MBs were used to get the SPMV as in Eqn SPMV pred _ mv MV%β where pred_mv is the MV predition o the urrent partition in sub-pixel resolution MV is the best integer-pixel MV o the urrent partition β4 in the /4-pixel ase and β8 in the /8-pixel ase and % represents the modulo operation. In [8] a larger partition MV e.g. 6x8 inter-mode MV takes a 6x6 MV as a reerene or previous rame MV was used to get the SPMV. I ombined with the SPMV rom CBFPS the auray o the SPMV an be greatly inreased. A more popular way to get the SPMV is to use a untion in most ases a seond-order untion to model the SAD surae [7 9]. I the mathing errors o the best integer-pixel MV and its 3

4 4 neighboring positions are known the oeiients o the untion an be solved. The position that orresponds to the smallest value in the SAD surae is then hosen as the SPMV. Many untions an be used to model the SAD surae. Example seond-order untions are listed as Eqns and xy x xy y x y y y x x y x where x and y are oordinates o the surae and x y is the mathing error SAD value. Normally the best integer-pixel position is set to be loated at so its neighboring integer-pixel positions are at - - et. As the number o model untion oeiients inreases more integer-pixel neighboring SADs are needed. In [7 9] Eqn 3 was used to determine one o the SPMVs whih used the best integer-pixel SAD and the SADs o its our diamond integer neighbors. Given these SAD values the oeiients o Eqn 3 an be omputed. The SPMV an then be alulated as: arg min C D A B y x y x SPMV y x p p 4 where L K D L K C J I B J I A / / / / L K J I and SAD I x p y p is a rational vetor its omponents are quantized into quarter-pixel units. Furthermore Xu et al. [7] proposed to use early terminations to urther redue the searh points rom the CBFPS method.

5 III. Analysis on Reduing Sub-pixel Searh Points with Multiple Partitions As shown in Setion II most previous ast sub-pixel ME methods redue the number o searh points by only searhing the redued area around the SPMV. For H.64/AVC multiple partition sizes they attempt to ind the best sub-pixel MV with the smallest SAD or eah partition beore the partition seletion as shown in Fig. a. However in pratie only the best partition o the MB needs preise sub-pixel MVs. The MVs o other partitions are only used or the inter-mode seletion. They are no longer useul ater the best partition is seleted. I a sub-pixel SAD is good enough to selet the best partition there is no need to searh or more preise sub-pixel points in the irst stage. Thereore i only a rough sub-pixel motion searh is perormed or eah partition the resulting MV does not neessarily have the smallest SAD and a preise sub-pixel MV is determined only or the best partition seleted then the number o searh points or the non-best partitions an be redued greatly. As shown in Fig. b the purpose o the irst stage ME is to obtain a rough sub-pixel SAD whih is lose to the best SAD. The integer-pixel SAD surae inormation an be used to deide whether the sub-pixel SAD is lose to the best one or not. Based on the above disussion we propose a new Rough-strategy-based Fast Sub-pixel Motion Estimation algorithm RFSME desribed in detail in the next setion. Sub-pixel motion searh or eah partition Compare and selet the best partition Stage : Rough sub-pixel motion searh or eah partition a Compare and selet the best partition Stage : Find the preise sub-pixel MV or the best partition b Fig. Fast sub-pixel ME approahes. a Proess or previous ast sub-pixel ME methods. b Proposed ast sub-pixel ME proess. 5

6 Step : Test the latness o the integer-pixel COST surae Yes Flat? No Step : Searh two sub-pixel points predited by two SPMVs; ompare the resulting COST with the integer COST Dierene Large? No Yes Step 3: Searh two more points vertially and horizontally next to the best point Step 4: Selet the best partition Step 5: Only or the best partition: small area searh around the best sub-pixel MV Fig. The proposed Rough-strategy-based Fast Sub-pixel ME algorithm. IV. The Fast Sub-pixel ME Algorithm The entire proess o the proposed Rough-strategy-based Fast Sub-pixel Motion Estimation RFSME algorithm an be desribed in Fig.. In our algorithm instead o using only the SAD to model the surae we use COST [3 ] as the ME mathing ost in the rest o the paper. The COST [3 ] is deined as: COST SAD λ RMV 5 MOTION where RMV is the number o bits to ode the MV and λ MOTION is the Lagrange multiplier []. λ MOTION is introdued to balane the importane between SAD and RMV. Note that COST an be viewed as a predition o the total number o bits or oding both the mathing error i.e. SAD and the orresponding side inormation i.e. MV. 6

7 In Step the dierene between the best COST o the integer position and the two averaged COSTs o its 4 neighboring integer positions the averaged COST o two vertial neighboring integer positions and the averaged COST o two horizontal neighboring integer positions are heked. I the dierene is small it means that the COST surae is quite lat and the best integer COST is lose to the optimal sub-pixel COST and thereore is good enough to estimate the best sub-pixel COST. In this ase the sub-pixel motion estimation is skipped or the urrent partition. The best COST o the integer position is used in the partition seletion in Step 4. The rule or deiding the COST surae latness is shown in Eqn 6 Not _ Flat i any o abis true COST _ Surae 6 Flat otherwise where the onditions a b and are a avg_ COST vertial > r COST or avg_ COST F ull horizontal > r COST F ull b i bloktype i min COST avg _ COST COST avg _ COST > th i bloktype ii ull vertial ull horizontal min COST avg _ COST COST avg _ COST > th ull vertial ull horizontal COST ull is the best COST ater ull-pixel ME avg_cost vertial is the COST average o its two vertial ull-pixel neighbors and avg_cost horizontal is the COST average o its two horizontal ull-pixel neighbors. r F is a ratio parameter to deide whether avg_cost vertial or avg_cost horizontal is lose to COST ull. bloktypei represents and 4 4 partitions and bloktypeii represents and 8 6 partitions. th and th are two thresholds. In the experiment o this paper th th and r F are set to and 5/4 respetively. These values are seleted based on the experimental statistis. I the COST surae is not lat in Step in Step two sub-pixel MV predition methods are used to get two SPMVs. The irst SPMV is alulated by the CBFPS method disussed in Setion II i.e. Eqn. The seond SPMV is alulated by the seond-order surae model disussed in Setion II. Ater these two points are searhed the point that has the smallest COST is seleted namely COST step. The motion vetor that orresponds to COST step is deined as MV step. 7

8 Table The distribution o absolute distane between the best sub-pixel MV x y and MV step x y Note: d x -x y -y in quarter-pixel units Sequene d< % d< % d< % News_QCIF Foreman_QCIF Mobile_QCIF Table lists the distribution o absolute distane d x -x y -y between the best sub-pixel x y MV and x y MV step the predited MV orresponding to COST step. The test ondition is the same as that desribed in Setion V. It shows that MV step an provide a good predition o the best sub-pixel MV. For example we an see rom Table that more than 7% MV step is exatly the same as the best sub-pixel MV and more than 94% MV step is within quarter-pixel distane rom the best sub-pixel MV. Thereore ater Step the assumption is made that MV step is lose to the best sub-pixel MV but COST step is not neessarily lose to the best sub-pixel COST. The absolute dierene between COST step and the best integer-pixel COST in Step COST best_ull_pixel is heked i.e. D COST step COST best_ull_pixel. I D is small this means that the COST doesn t derease muh between COST step and the best integer-pixel COST and that COST step is already lose to the best sub-pixel COST and is good enough or the mode seletion. In this ase COST step is used in the partition seletion in Step 4. The rule or deiding whether D is small or not an be desribed in Eqn 7 D is Large i any o ab is true Small otherwise 7 where a avg _ COST vertial > r COST or avg _ COST D min_step horizontal > r COST D min_step b i bloktype i D > th i bloktype ii D > th COST min_ step min COSTstep COST avg_cost vertial and avg_cost horizontal are the same as in best _ ull _ pixel Eqn 5. r D is a ratio parameter to deide whether avg_cost vertial or avg_cost horizontal is lose to COST min_step and it is set to 3/ in this paper. 8

9 I D is large COST step may not be lose to the best sub-pixel COST as shown in Fig. 3a. In this ase the two points vertially and the two points horizontally next to MV step in quarter-pixel resolution will be heked. As shown in Fig. 3b the blak point is MV step the grey points are quarter-pixel neighbors o MV step and the white points are integer neighboring points o MV step. In Step 3 two searh points are seleted as one point out o V and V and one point out o H and H. A bilinear model as desribed below is used to selet one o the neighboring points. As shown in Fig. 4 a the slopes are irst omputed based on Eqn 9 between the two horizontal neighboring integer points or the two vertial neighboring integer points and the best sub-pixel point rom Step the point by MV step. Then the quarter-pixel neighboring point is seleted orresponding to the slope with the smaller slope value as shown in Fig. 4 b and Eqn 8. < < Horizontal H i SH SH V i SV S Vertial V P step3 and PStep3 8 H i SH > SH V i SV > SV where S COSTinteger _ i COSTmin_step i V V H H 9 Coord Coord i integer _ i min_step integer_i represents the losest integer-pixel point in i s diretion i.e. V and V or the vertial diretion and H and H or the horizontal diretion and min_step represents the best sub-pixel point ater Step. Coord is the oordinate in quarter-pixel resolution o the points. The X-oordinate horizontal diretion is used or H and H and the Y-oordinate vertial diretion is used or V and V. integer_v integer_h V integer_h 9 H integer_v V H MV step a b Fig. 3 a An example COST surae or COST step not lose to the best sub-pixel COST and b MV step and its quarter-pixel neighboring points. Y X

10 integer_h or integer_v MV step The neighboring sub -pixel point seleted X or Y integer_h or integer_v integer_h or integer_v H or V MV step Step H or V X or Y integer_h or integer_v H or V H or V a b Fig. 4 Using the bilinear model to selet neighboring searh points white points: integer pixel; blak points: MV step ; grey points: neighboring point seleted. Note: in a the let slope is smaller than the right slope. Thereore in b the neighboring sub-pixel point on the let is seleted. Ater Steps and 3 a COST value COST rough an be obtained or eah partition whih is lose or equal to the best COST. The sub-pixel MV that orresponds to COST rough is denoted by MV rough. In Step 4 COST rough is used to selet the best partition. In Step 5 a small area sub-pixel reinement is perormed around MV rough. In the proposed algorithm the eight quarter-pixel neighbors around MV rough are searhed. Sine Step 5 is perormed only or the best partition seleted the average searh points per partition is redued ompared to onventional ast sub-pixel searh algorithms. It should be noted that the proposed RFSME algorithm is just one implementation o our idea desribed in Setion III. Our method is general and it ould also be implemented in other ways. For example we an simply skip the sub-pixel searh in the rough searh step and diretly use the best ull-pixel searhing results to selet the partition and then perorm the preise searh or the best partition. This an be viewed as a simpliied version or extension o the RFSME algorithm. V. Experimental Results We implemented our proposed algorithm on the H.64/AVC reerene sotware JM [5]. In the experiments eah test sequene o rames is oded. The piture oding type is IPPP and the

11 rame rate is 3 rames/se. The searh range is 6 or QCIF and 3 or CIF and Standard Deinition SD. The number o reerene rames is one. Full searh is used or the integer pixel ME in our experiment [5]. It should be noted that our algorithm is general and various other integer pixel ME algorithms an also be easily implemented as will be disussed later. Six methods are ompared or eah sequene: JM Reerene Method [5] Sub-pixel Full Searh The method in [6] CBFPS 3 The method in [7] FPME 4 The method in [8] PDFPS 5 Use the best Integer COST diretly to selet the partition and then use JM s method to perorm the sub-pixel ME or the best partition IESME-Proposed. As mentioned this method an be viewed as the simpliied version or extension o the proposed RFSME algorithm. 6 The Proposed RFSME Method RFSME-Proposed In Table the Peak Signal to Noise Ratio PSNR bitrate BR and average searh points SP per partition size SP/PT [3 7] or eah method are ompared or sequenes in dierent resolutions and with dierent quantization parameters QPs. The rate-distortion R-D urves or some sequenes in Table are shown in Fig. 5 a and b. Furthermore Fig. 5 and d shows the BR-SP/PT urves or dierent methods. Table Comparison o dierent ME methods Sequene Method PSNR db BR kbps SP/PT Akiyo QCIF 76x44 QP4 Mobile QCIF 76x44 QP8 Full Searh CBFPS FPME PDFPS IESME-Proposed RFSME-Proposed Full Searh CBFPS FPME PDFPS IESME-Proposed RFSME-Proposed

12 Football CIF 35x88 QP8 Football CIF 35x88 QP8 Mobile SD 7x576 QP8 Flower SD 7x576 QP4 Full Searh CBFPS FPME PDFPS IESME-Proposed RFSME-Proposed Full Searh CBFPS FPME PDFPS IESME-Proposed RFSME-Proposed Full Searh CBFPS FPME PDFPS IESME-Proposed RFSME-Proposed Full Searh CBFPS FPME PDFPS IESME-Proposed RFSME-Proposed a Akiyo QCIF b Football CIF Foreman_QCIF d Akiyo_QCIF Fig. 5 R-D and BR-SP/PT urves omparisons or dierent methods a and b: R-D urves and d: BR-SP/PT urves

13 Several observations an be drawn rom Table and Fig. 5: The previous methods CBFPS FPME and PDFPS an redue the SP by reduing the searh area around the predited PMV. However our proposed methods IESME-proposed and RFSME-proposed an urther redue more than hal the SP ompared to previous methods CBFPS FPME and PDFPS by only perorming the preise searh on the best partition. The IESME-proposed method an redue the most number o searh points but the perormane derease is also large or some sequenes e.g. or Akiyo_QCIF in Fig. 5-a. This implies that only using the best integer-pixel COST may not always be able to ind the best partition mode suitably and some sub-pixel motion searh may be needed to help selet the best mode. However due to its smallest number o SPs the IESME-proposed method an still be very useul in situations where omputation omplexity is a ruial ator and some quality degradation is tolerable. The RFSME-proposed method has the best overall perormane. Compared with FS and other previous methods CBFPS FPME and PDFPS the RFSME-proposed method has muh smaller SP while keeping almost the same oding perormane. Compared with the IESME-proposed method the proposed method has obviously better oding perormane. With the RFSME-proposed method the SP per partition size an be redued to less than 3 or most sequenes. The SP an be urther redued and beomes lose to that o the IESME-proposed method when oding low motion videos suh as Akiyo_QCIF. When QP dereases the SP-per-partition-size or most o the methods will slightly inrease. This is beause a the reovered reerene rames are more preise or smaller QPs i.e. higher PSNR. Thereore the hane or the MB to selet a smaller partition size beomes higher. b When the reerene rames are more preise the COST surae or the interpolated sub-pixel loations may beome more omplex and it may take more steps to ind the best sub-pixel loation. Besides the above observations there are also other advantages o the proposed method. Firstly the proposed RFSME algorithm models the sub-pixel COST surae based on the 4-neighboring integer COST values. Thus the algorithm an be easily ombined with most o the existing ast integer ME algorithms. Sine most ast integer ME algorithms [4 4-6] e.g. Simpliied Hex 3

14 Searh [4] and Diamond Searh [5] end the ME proess by searhing the 4-neighboring points around the best integer point using the 4-neighbor COST inormation does not introdue any extra ost to the integer ME proess. Furthermore with the development o new video oding standards suh as High Eiieny Video Coding HEVC and Next Generation Video Coding NGVC [] some existing sub-pixel ME methods may no longer work. For example with the introdution o Adaptive Interpolation Filter [3] in HEVC or NGVC the seond-order sub-pixel COST surae model may beome unsuitable sine the interpolation ilter will adapt to the rame ontents. This will greatly limit the useulness o many ast sub-pixel methods [7 9] whih rely on this seond-order model. Compared to these methods our proposed methods an still work eiiently ater some simple extensions. This is beause a the basi idea o our method is to redue sub-pixel SP by perorming rough searh in the non-best partitions. As long as we an ind some way to perorm rough searh the proposed method an be easily applied to the new standards. b There may be more partition sizes introdued in the uture standards. With the introdution o more partition sizes our proposed method an work even more eiiently by reduing the sub-pixel SP in the non-best partitions. Table 3 shows another experiment or a multiple reerene rame ase. In this ase our algorithm irst perorms the rough searh or all partitions on all reerene rames and then perorms the preise searh only or the best partition on the best reerene rame. From Table 3 we an see that our algorithm an urther redue SP/PT by perorming the rough searh on those non-best reerene rames. Table 3 Results or Football CIF using 3 reerene rames with QP8 Full Searh CBFPS FPME PDFPS IESME RFSME PSNR db BR kbps SP/PT VI. Conlusion In this paper a ast and eetive sub-pixel ME algorithm is proposed. It redues the number o average SP per partition by more than hal ompared to onventional ast sub-pixel ME algorithms 4

15 with relatively small perormane dereases. Aknowledgements The main part o this work was perormed when employed at Motorola. This paper is also supported in part by the ollowing grants: Chinese national 973 grants CB734 and CB7346 National Siene Foundation o China grants and Reerenes [] T. Wiegand G.J. Sullivan G. Bjontegaard and A. Luthra Overview o the H.64/AVC video oding standard IEEE Trans. Ciruit System Video Tehnology vol. 3 pp [] J. Zhang and Y. He Perormane and omplexity joint optimization or H.64 video oding IEEE Int l Symp. Ciruits and Systems vol. pp [3] W. Lin D. M. Baylon K. Panusopone and M.-T. Sun Fast sub-pixel motion estimation and mode deision or H.64 IEEE Int l Symp. Ciruits and Systems pp [4] Z. Zhou and M.T. Sun Fast maroblok inter mode deision and motion estimation or H. 64/MPEG-4 AVC Int l Con. Image Proessing vol. pp [5] JM. [6] Z. Chen P. Zhou and Y. He Fast integer-pixel and rational pel motion estimation or JVT JVT-7 6th Meeting: Awaji Island JP. [7] J.F. Chang and J.J. Leou A quadrati predition based rational-pixel motion estimation algorithm or H.64 IEEE Int l Symp. Multimedia pp [8] L. Yang K. Yu J. Li and S. Li Predition-based diretional rational pixel motion estimation or H.64 video oding IEEE Int l Con. Aoustis Speeh and Signal Proessing vol. pp [9] J.W. Suh and J. Jehang Fast sub-pixel motion estimation tehniques having lower omputational omplexity IEEE Trans. Consumer Eletronis vol. 5 pp [] W. Lin K. Panusopone D. M. Baylon M.-T. Sun A new lass-based early termination method or ast motion estimation in video oding IEEE Int l Symp. Ciruits and Systems pp [] T. Weigand H. Shwarz A. Joh F. Kossentini and G. Sullivan "Rate-Constrained Coder Control and Comparison o Video Coding Standards" IEEE Trans. Ciruits Syst. Video Tehnol. vol. 3 no. 7 pp [] Douments o the Joint Collaborative Team on Video Coding [3] Y. Vatis J. Ostermann Adaptive interpolation ilter or H.64/AVC IEEE Trans. Cir. Syst. Video Tehnol. vol. 9 no. pp [4] X. Yi J. Zhang N. Ling and W. Shang Improved and simpliied ast motion estimation or JM JVT-P Poznan Poland 4-9 July 5. [5] S. Zhu and K.-K. Ma A new diamond searh algorithm or ast blok mathing motion estimation IEEE Trans. Image Proessing vol. 9 no. pp [6] H.-Y. C. Tourapis and A. M. Tourapis Fast motion estimation within the H.64 ode Int l Con. Multimedia & Expo vol. 3 pp [7] X. Xu and Y. He Improvements on ast motion estimation strategy or H.64/AVC IEEE Trans. Ciruit System Video Tehnology vol. 8 pp