Signal Processing: Image Communication

Transcription

1 Signal Processing: Image Communication 7 () Contents lists available at SciVerse ScienceDirect Signal Processing: Image Communication journal homepage: motion estimation scheme for secondary SP-frame coding using inter-frame correlation and FMO Ki-Kit Lai, Yui-Lam Chan n, Chang-Hong Fu, Wan-Chi Siu Centre for Signal Processing, Department of Electronic and Information Engineering, he Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong article info Article history: Received January Accepted 8 August Available online September Keywords: SP-frame Bitstream switching FMO QDC domain Motion estimation abstract o cope with considerable size of secondary SP-frames, quantized-transform domain motion estimation has recently been proved to be appropriate for the coding of secondary SP-frames in H./AVC. Nevertheless, its computational complexity is tremendous and there are still some situations that pixel-domain motion estimation can perform better. Both techniques are therefore not implemented solely in secondary SP-frame coding. In this paper, a hybrid scheme is proposed to effectively combine two existing motion estimation techniques. he combination is based on a new measurement of inter-frame correlation using the bit-counts of the macroblocks in SP-frames, so that the hybrid scheme is dominated by employing quantized-transform domain motion estimation in the macroblocks with weaker inter-frame correlation; otherwise, it approaches to pixel-domain motion estimation. With the further help of the explicit mode in Flexible Macroblock Ordering (FMO), the proposed hybrid scheme classifies MBs into two slice groups by examining the domain used in motion estimation prior to coding motion vectors in a secondary SP-frame. he slice structure of a secondary SP-frame using the explicit FMO mode is flexible and can be changed during the encoding of each new frame. Simulation results show that our proposed scheme overwhelmingly outperforms the quantized-transform domain motion estimation scheme. As a consequence, the size of secondary SP-frames can be reduced remarkably with significant computational reduction. & Elsevier B.V. All rights reserved.. Introduction SP-frame [,] is a new picture type of the H./AVC Extended Profile that can be perfectly reconstructed using different reference frames. his feature enables seamless video streaming in heterogeneous networks where bit rate adaption is required. In this scenario, multiple bitstreams encoded at different bit rates are stored in a server to deal with network bandwidth variation. Switching among multiple bitstreams can be accomplished by inserting SP-frames in the bitstreams. It is well known n Corresponding author. el.: þ8 7; fax: þ address: enylchan@polyu.edu.hk (Y.-L. Chan). that I-frames can also be used for this purpose because no temporal prediction is included. However, periodic I-frame insertion will sacrifice the coding efficiency in order to support rapid switching among multiple bitstreams. SP-frames then offer an attractive alternative for seamless bitstream switching. he SP-frame scheme in H./AVC is composed of primary and secondary SP-frames. hey both exploit temporal redundancy with predictive coding, but use different reference frames. Although different reference frames are used, it still allows an identical reconstruction. his property allows the use of SP-frames to replace I-frames for drift-free switching among multiple compressed bitstreams. Besides, the coding efficiency of a primary SP-frame is much better than that of an I-frame and is slightly worse than that of a P-frame [ ]. Nevertheless, extra storage for 9-9/$ - see front matter & Elsevier B.V. All rights reserved. doi:./j.image..8.

2 K.-K. Lai et al. / Signal Processing: Image Communication 7 () secondary SP-frames is inevitably required. In [ ], some investigations were conducted to evaluate the trade-off between the coding efficiency of primary SP-frames and the storage cost of secondary SP-frames for multiple bit rate video applications. It is found that a primary SP-frame with high quality results in a significantly high storage requirement for a secondary SP-frame. an et al. [] revealed that the bulky size of a secondary SP-frame can be reduced by avoiding the reuse of coding modes and motion vectors of a primary SP-frame during secondary SP-frame coding. hus a method was proposed to improve the coding efficiency by correctly choosing reference pictures. When switching is performed from bitstream (B) to bitstream (B) as shown in Fig., the motion vectors and modes are calculated between the reconstructed reference frame in B (P,t- ) and the reconstructed target frame in B being switched to (SP,t ). But, this coding arrangement can only reduce the size of a secondary SP-frame in multiple bit rate switching scenarios by about %. Meanwhile, Lai et al. [ 8] disclosed the problem of using the traditional pixel-domain motion estimation and compensation when secondary SP-frames are employed in bitstream switching. An analysis in [ 8] shows that the coding mechanism of secondary SP-frames does not pave the way for entropy coding. Macroblocks in secondary SP-frames involve the transformation and quantization processes prior to calculating the residue. Many non-zero coefficients need to be encoded without further quantization as the conventional encoder. Quantized-transform (QDC) domain motion estimation and compensation processes adopting in secondary SP-frame coding were therefore proposed and can compensate the demerit of their pixel-domain counterpart. Nonetheless, computational complexity is tremendous and there are still some exceptional cases that the coding performance of the QDC-domain technique is not as good as that of the pixel-domain technique. A hybrid motion estimation scheme for secondary SP-frame coding based on interframe correlation is proposed in this paper. With the help of flexible macroblock order in H./AVC, the hybrid scheme effectively combines two existing techniques, operated in different domains, to further reduce the size of secondary SP-frames. he rest of the paper is organized as follows. Section gives a brief description of the conventional SP-frame coding process and the QDC-domain motion estimation algorithm. An in-depth study of the problem on applying both motion estimation techniques into a secondary SP-frame encoder is provided in Section. Analysis of relationship between primary and secondary SP-frames is also covered in this section. After the detailed investigation, a novel hybrid scheme with the support of Flexible Macroblock Ordering (FMO) is then proposed. In Section, experimental results are shown, which are focused on the comparison between our proposed scheme and the QDC-domain scheme employed in the secondary SP-frame encoder. Finally, the conclusions of the paper are presented in Section.. Review of SP-frame coding An example of bitstream switching using SP-frames is shown in Fig.. Inthisfigure,animagesequenceisencoded into two bitstreams (B and B) with different bit rates using two quantization parameters, and, respectively. wo primary SP-frames SP,t and SP,t are placed at frame t (switching point) within each bitstream. In addition, a secondary SP-frame (SP,t ) is produced, which has the same reconstructed values as SP,t even though different reference frames are used. When switching is needed from B to B at frame t, a secondary SP-frame (SP,t )insteadofsp,t is transmitted. he decoder can obtain seamlessly as normally SP,t decoded at frame t. Consequently, it can continually decode frame tþ from B seamlessly. he block diagram of a primary SP-frame encoder is depicted in the upper part of Fig.. In comparison with a P-frame encoder, the only difference of a primary SP-frame encoder is the extra quantization/dequantization steps with a quantization level Qs applied to the transform coefficients of the primary SP-frame (SP,t in Fig. ) [,]. Interested readers are encouraged to read [,]. hese extra steps ensure that the quantized-transform coefficients of SP,t are divisible by Qs, which is used in the encoding process of the secondary SP-frame, SP,t, as shown in the middle part of Fig.. For encoding SP,t, the prediction of the reference frame is firstly transformed and quantized using Qs before generating the residue. B mv mode P,t- P,t- SP,t P,t+ P,t+ mv mode S P,t B ime P,t- P,t- SP,t P,t+ P,t+ mv mode t- t- t t+ t+ Fig.. Switching bitstream from B to B using SP-frames.

3 K.-K. Lai et al. / Signal Processing: Image Communication 7 () B + QP - mv, mode QP - VLC + Bitstream B MC ME Pˆ Frame Buffer, t Primary SP-frame Loop Filter - Qs - Qs mv, mode Qs + - VLC Bitstream SP Secondary SP-frame B + QP - mv, mode QP - VLC + Bitstream B MC Primary SP-frame Qs [[SP,t ]] ME Frame Buffer Loop Filter - Qs - Qs Fig.. Simplified encoding block diagram of primary and secondary SP-frames [,]. Both the prediction and the quantized-transform coefficients of SP,t in the lower part of Fig. are thus synchronized to Qs. From this point, there is no further quantization. It means that the decoder can perfectly reconstruct the quantized-transform coefficients of SP,t. Producing secondary SP-frames also involves motion estimation independently in all modes and submodes of H./AVC [9 ] by minimizing the Lagrangian cost function J motion : J motion ðmv,l motion Þ¼SADðB c,b r Þþl motion R motion ðmv pmv Þ where mv is the motion vector, pmv is the motion vector predictor, l motion is the Lagrangian multiplier for motion estimation, R motion (mv pmv ) is the estimated number of bits for coding mv, and SAD is the sum of absolute differences between the current block B c and its reference block B r [,]. After the motion estimation for each mode, a rate-distortion (RD) optimization technique is used to obtain the best mode and its general equation is given by J mode ðb c,b rec, mode,l mode Þ¼SSDðB c,b rec, mode Þ þl mode R mode ðb c,b rec, mode Þ ðþ where l mode is the Lagrangian multiplier for mode decision, mode is one of the candidate modes during motion estimation, SSD is the sum of squared differences between ðþ B c and its reconstruction block B rec, and R mode (B c,b rec, mode ) represents the number of coding bits associated with the chosen mode. o compute J mode, forward and inverse integer transforms and variable length coding are performed. However, there is a deficiency in using the conventional motion estimation and compensation processes, which are operated in pixel domain, for secondary SP-frames. In H./AVC [], the motion estimation process in P-frames and secondary SP-frames is the same. For P-frame coding, the best match in the reference frame is subtracted from the current macroblock to form a residual macroblock. hen the transformation and quantization processes are performed on the residue. Most of the coefficients become zero after these processes. his property paves the way for entropy coding. On the other hand, secondary SP-frame coding involves firstly the transformation and quantization processes of the macroblock in SP,t and its best match in the reference frame, SP,t-. hey are then subtracted in quantized-transform (QDC) domain. In this case, their quantized-transform coefficients are only near, but not equal. here are many nonzero coefficients generated, especially for a small Qs. Since there is no further quantization from this point, the spread of non-zero coefficients in entropy coding exists. his induces the bulky size of a secondary SP-frame. o reduce residues for better entropy coding, quantized-transform domain motion estimation ()

4 K.-K. Lai et al. / Signal Processing: Image Communication 7 () B mv,mode + QP + - VLC Bitstream B MC ME Frame Buffer Pˆ, t Qs QDC-MC mv,mode - + VLC Bitstream SP B mv,mode QP - + VLC Bitstream B MC ME Frame Buffer Qs[[SP,t ]] Qs Fig.. Simplified secondary SP-frame encoder using. instead of its pixel-domain counterpart is used [ 8]. Fig. shows the SP-frame encoder implemented with. Instead of pixel-domain, the aim of is to minimize residues in QDC domain. herefore, the modified Lagrangian cost function of, J motion,is based on the sum of absolute differences in quantizedtransform coefficients and the estimated rate. his cost function is expressed as Jmotion ðmv,l motion Þ¼k SAQDðB c,b r Þþl motion R motion ðmv pmv Þ where SAQD(B c,b r ) denotes the sum of absolute differences between the quantized-transform coefficients of the current block B c and the quantized-transform coefficients of its reference block B r, and it can be defined as SAQDðB c,b r Þ¼ X 9Qs½ðB c ÞŠ Qs½ðB r ÞŠ9 ðþ k in () is a weighting factor to compensate for the energy loss of SAQD(B c,b r ) due to the fact that SAQD(B c,b r )is operated in the QDC domain and the extra quantization causes the energy is no longer preserved. Using this arrangement, the motion estimation is now consistent with the residue generation of the secondary SP-frame encoder. In summary, the pixel-domain motion estimation is to minimize the absolute differences in which the subtraction ðþ is performed in terms of pixels while the QDC-domain motion estimation is to minimize the absolute differences in which the subtraction is performed in terms of transformed and quantized coefficients. he is capable of finding the motion vectors with more zero values of quantized-transform coefficients, which benefits the entropy coding of a secondary SP-frame, and provides a remarkable size reduction [ 8].. Proposed hybrid secondary SP-frame coding.. Impact of using on secondary SP-frame coding In [7], it was found that some secondary SP-frames generated by the QDC-domain motion estimation technique induce more bits than the pixel-domain technique, as illustrated in Fig. (a) and (b) where the test video streams used for simulation are Riverbed and Shuttlestart. hey were encoded into two bitstreams with being equal to and 8, and Qs was set to, i.e. and, respectively. Switching was then taken place from the bitstream with ¼ to the bitstream with ¼8. his phenomenon can be explained by the distribution of quantized-transform coefficients in some blocks where all pixels are similar or with the same value.

5 K.-K. Lai et al. / Signal Processing: Image Communication 7 () Fig.. Frame-by-frame comparisons of size reduction of secondary SP-frames in percentage achieved by over pixel-domain ME in (a) Riverbed and (b) Shuttlestart. In these blocks, their quantized-transform coefficients are mainly zeros except the DC coefficient. If this type of QDC-domain blocks are used to search over a predetermined search area on the quantized-transform coefficients of the reference frame, all the SAQD values in () are very similar and the motion estimation process becomes very sensitive to noise. In this case, the smallest SAQD value may not be presumed to be the best motion vector, and then introduces more bit-counts. Another drawback of is the surge in computational complexity. For, each current block B c is transformed and quantized to Qs[(B c )]. A search window in the reference frame centered on the current block position is set. starts with transforming and quantizing a block, Qs[(B r )], in the top right-hand corner of the search window. After obtaining Qs[(B c )] and Qs[(B r )], SAQD(B c,b r ) can then be calculated for this candidate. Afterward, the next candidate is another block shifted by pixel in horizontal direction, and this block is also transformed and quantized for computing its SAQD(B c,b r ). hese procedures continue for all possible candidates within the search window. It means that all possible candidates within the search window also need to be transformed and quantized in QDC domain. Consequently, SAQD is computationally very intensive though it can achieve higher coding efficiency as compared with SAD... MB-based hybrid motion estimation scheme he aforementioned drawbacks motivate us to adopt a hybrid approach to reduce the computational complexity and further improve the coding efficiency of. In the proposed hybrid scheme, a selection mechanism based on inter-frame correlation between the current frame and the reference frame is adopted to choose between the use of pixel-domain and QDC-domain techniques. One straightforward approach is to perform pixel-domain motion estimation and QDC-domain motion estimation separately. his frame-based approach compares the sizes of secondary SP-frames generated by both estimation techniques and chooses the final motion vector with a smaller bit-count. However, spatial characteristic within a frame is not considered. For instance, a block in homogeneous area causes more quantizedtransform coefficients to be zero in which is no longer suitable. In contrast, choosing spatial-domain ME might give a lower-energy residual after motion compensation in this situation. In general, spatial-domain ME is appropriate for homogeneous areas of the frame while is beneficial to detailed areas. By taking this into consideration, the scheme proposed in this paper is operated at macroblock (MB) level. Similar to the frame-based approach, two independent motion estimation techniques operated in pixel and QDC domains for each MB are carried out. hen two bit-counts for an MB can be given and the minimum one, associated with a motion vector, is chosen. his motion vector is considered as the best one. However, this brute-force approach will increase the encoding time drastically. he results given in [7] revealed that the improvement in the coding efficiency of a secondary SP-frame from highly relies on the degree of inter-frame correlation between the MB in SP,t and its motion compensation MB in P,t- as shown in Fig., and it is denoted by corr SSPMB,t. he weaker the correlation, the better the coding efficiency of. herefore, corr SSPMB,t is a good measure to determine a proper domain for motion estimation in coding an MB in a secondary SP-frame. In other words, a smaller value of corr SSPMB,t tends to use ; otherwise, pixel-domain motion estimation is appropriate to reduce the required computational complexity and maintain sufficient coding efficiency for MBs in homogeneous areas. he number of bits required for encoding the MB in SP,t can be used directly as a measure of corr SSPMB,t. However, this bit-count cannot be obtained prior to secondary SP-frame coding. Fig. (a) and (b) then shows the bit-counts of all MBs in two rows of SP,t and SP,t in the 8nd frame of Riverbed and the 9th frame of Shuttlestart. It is noted that the MB of the primary SP-frame has already been available when its corresponding MB in the secondary SP-frame is encoded. From this figure, it can be easily seen that the general trends of the two curves are very similar. It is due to the fact that the current frames used for encoding SP,t and SP,t are the frames at time t from B and B, respectively. hey are actually the same video content and the only discrepancy is the quantization parameter. It implies that corr SSPMB,t is reasonably approximated by the bit-count of the MB in the

6 K.-K. Lai et al. / Signal Processing: Image Communication 7 () Bits 8 th Row primary SP secondary SP 7 Position of macroblocks in the th row Bits 8 th Row primary SP secondary SP 7 Position of macroblocks in the th row 8 pixels 7 pixels Bits 8 th Row primary SP secondary SP 7 Position of macroblocks in the th row Bits 8 8 pixels th Row primary SP secondary SP 7 Position of macroblocks in the th row 7 pixels Fig.. Bit-counts for all macroblocks in two rows of a secondary SP-frame and its corresponding primary SP-frame in (a) Riverbed and (b) Shuttlestart.

7 K.-K. Lai et al. / Signal Processing: Image Communication 7 () 7 primary SP-frame at time t, SP,t,denotedbybit-count PSPMB. It is then used to select a proper domain for motion estimation in secondary SP-frame coding with the proposed hybrid scheme. When bit-count PSPMB is larger than a predefined threshold H, corr SSPMB,t becomes lower and is carried out for secondary SP-frame coding in order to offer higher coding efficiency. In contrast, pixeldomain motion estimation is good enough when bitcount PSPMB rh, which can relieve the computational burden of secondary SP-frame coding as well as providing better coding efficiency of MBs within the homogeneous area. In general, more MBs in a frame with complex spatial or temporal activities will be encoded using. o determine H, arithmetic mean for each frame with M N MBs is used as a tool to derive the central tendency of bits in these MBs. H can then be formulated as H ¼ XM XN bit count MN PSPMBði,jÞ ðþ i ¼ j ¼ where bit-count PSPMB(i,j) is the bit-count needed to be encoded for the MB at the ith row and jth column of a primary SP-frame... Utilization of FMO in the proposed hybrid scheme he proposed hybrid scheme estimates motion vectors obtained from two kinds of domains pixel domain or QDC domain. Unsystematically mixing these two kinds of motion vectors might increase the bit-count required by a secondary SP-frame. It is due to the adoption of motion vector prediction where motion vectors are coded differentially with respect to a predictor of the motion vectors from the previous coded blocks. his predictor is computed with consideration of the motion vectors of the adjacent blocks. hey tend to have very high correlation. As a consequence, the motion vector predictor is formed based on the median value of the motion vectors of the three adjacent blocks, on the left, top, and top-right (or top-left if top-right is not available). However, considering that the proposed hybrid scheme allows the use of different domains for motion estimation in secondary SP-frame coding, the motion vector of the MB with fewer bit-count obtained by ME in one domain will contribute to the motion vector predictor for the adjacent MB, no matter which domain is used in motion estimation [,]. Motion vectors obtained from different domains are spatially less correlated, resulting in an increase in bitcounts of secondary SP-frames. It is noted that, in H./ AVC video, slice is a basic structure in a picture, and each picture can be subdivided into one or more slices. A sequence of MBs is defined in these slices. hese slices segment a picture into different partitions and they are coded independent of each other. In the typical picture setting of an H./AVC encoded video bitstream, only one slice is always used in a picture. In other words, even though two neighboring MBs in a slice using different motion estimation techniques are still dependent on each other, mixing two kinds of motion vectors in a frame is not good for coding efficiency. It means that only one slice per frame is not very applicable to the proposed hybrid scheme. his problem can be solved by assigning only one MB per slice. Yet it is not a practical way due to the excessive bits required by the headers in these slices. In the proposed hybrid scheme, an additional consideration is made based on the new MB classification policy by utilizing Flexible Macroblock Ordering (FMO) []. FMO is one of the most striking error resilience tools supported by H./AVC. It specifies a pattern that assigns MBs in a picture to one or several slice groups, and provides more flexible way of macroblock grouping. Each MB can be assigned into a slice group through a Foreground background ype ype ype ype ype Fig.. FMO map types. ype

8 8 K.-K. Lai et al. / Signal Processing: Image Communication 7 () Macroblock-to-slice Allocation map (MBAmap) [ 7]. In this map, each MB is identified with a number that indicates which slice group the MB belongs to. he MBAmap can update each frame using the Picture Parameter Set (PPS). By this mechanism, FMO can divide a picture with different patterns of MBs. here are seven different types of slice group maps for FMO in the standard [9]. Fig. depicts six predefined slice group maps from type to type : interleaved, dispersed, foreground and background, box-out, raster scan, and wipe. In addition, there is an explicit mode (type ), which is also the most flexible FMO type. his explicit mode allows users to define their own MBAmap such that MBs in a picture can be assigned to any slice group in any order. In the proposed hybrid scheme, motion estimation is carried out either in pixel domain or QDC domain. According to the domain selection in Section., there are two slice groups in each secondary SP-frame, and MBs are classified by examining the bit-counts of the primary SP-frame prior to the motion estimation and motion vector coding in a secondary SP-frame. he six predefined FMO slice groups in Fig. are not flexible enough to fulfill the classification of MBs since none of the patterns in the predefined slice groups can fit for all frames. Apart from the predefined patterns, fully flexible macroblock ordering in the explicit mode is used in the proposed scheme. By doing so, each secondary SP-frame changes dynamically the MB classification throughout the entire video sequence. he provision of dynamic formation of slice groups in every secondary SP-frame is exploited by the bit-counts of MBs in the primary SP-frame. hen each MB in the corresponding secondary SP-frame is classified either in pixel or QDC domain. Specifically, an identification number is given to each MB. he MBs that are identified using pixeldomain ME are assigned as slice group while the MBs that are classified using are assigned as slice group. After that, an MBAmap is established. his map partitions the PSNR (db) pixel-me [] pixel-me [] Bitrate (MBits/s) Bitrate (MBits/s) PSNR (db) PSNR (db) 9 8 pixel-me [] 7 pixel-me [] Bitrate (MBits/s) Bitrate (MBits/s) PSNR (db) PSNR (db) pixel-me [] PSNR (db) Bitrate (MBits/s) Bitrate (MBits/s) 9 8 pixel-me [] Fig. 7. Rate-distortion performance for switching-down scenario in (a) Riverbed, (b) Duckstakeoff, (c) Shuttlestart, (d) Crew, (e) Mobisode, and (f) Mobisode.

9 K.-K. Lai et al. / Signal Processing: Image Communication 7 () 9 secondary SP-frame into two slice groups. During encoding, only MBs in the same slice depend on each other. It means that motion vector predictors, which are the motion vectors computedthroughmotionestimationinthesamedomain, are from the same slice only. On the whole, FMO partitions a secondary SP-frame into two slice groups referring to the MBAmap, which describes the use of the proper domain in motion estimation of each MB.. Results A large amount of experimental work has been conducted to evaluate the performance of the proposed hybrid scheme for bitstream switching using SP-frames. Results in terms of both coding efficiency and computational complexity were compared with those obtained using the pixel-domain and QDC-domain schemes. Let us denote them as pixel-me [] and. All schemes were implemented based on the H./AVC reference software (JM version.) [] for secondary SP-frame coding. Six test sequences, Riverbed (8 7 pixels), Duckstakeoff (8 7 pixels), Shuttlestart (8 7 pixels), Crew (8 7 pixels), Mobisode (8 8 pixels) and Mobisode (8 8 pixels), were used for performance comparison. In each sequence, frames were encoded with two Q P to generate two bitstreams with Fig. 8. Frame-by-frame size reduction of secondary SP-frames in percentage achieved by the proposed hybrid scheme and over pixel-me [] for the switching-down scenario in (a) Riverbed, (b) Duckstakeoff, (c) Shuttlestart, (d) Crew, (e) Mobisode, and (f) Mobisode.

10 K.-K. Lai et al. / Signal Processing: Image Communication 7 () different bit rates. Switching between two bitstreams in both directions was then performed. For the high bit rate bitstream, the quantization parameter Q P was fixed at. On the other hand, Q P was varied from to with a step size of for coding the low bit rate bitstream. According to the optimal setting in [], Q S was set to Q P, i.e. from 8 to. Only the first frames of the bitstreams were encoded as I-frames, and switching frames were encoded in turn as SPframes while all the rest non-switching frames were encoded as P-frames. In our experiments, the extended profile with CAVLC entropy encoding was used to configure the encoder. R-D optimization was enabled. For the motion estimation process, a search range of was set for both P-frames and SP-frames. It is based on the encoding conditions recommended in [8]. For the quantized-transform motion estimation, the scaling factor k of Eq. () was set to, which was found by experimental observations [7]. o make the comparison impartial, both schemes employed a full search motion estimation algorithm. We aim at evaluating the coding efficiency of secondary SP-frames based on our proposed hybrid scheme. o have a comprehensive and fair comparison among the proposed hybrid scheme, and pixel-me, we did exhaustive simulation on all possible secondary SPframes. Fig. 7 shows the rate-distortion curves of secondary SP-frames using pixel-me,, and the hybrid scheme for switching-down scenario. It is noted that the PSNR values of secondary SP-frames encoded by three Fig. 9. Frame-by-frame size reduction of secondary SP-frames in percentage achieved by the proposed hybrid scheme and over pixel-me [] for the switching-up scenario in (a) Riverbed, (b) Duckstakeoff, (c) Shuttlestart, (d) Crew, (e) Mobisode, and (f) Mobisode.

11 K.-K. Lai et al. / Signal Processing: Image Communication 7 () different schemes are identical for the same Q p. It is because the reconstructed secondary SP-frame is exactly the same as its corresponding primary SP-frame no matter which motion estimation approach is used in secondary SP-frame coding. From Fig. 7, it is obvious that the proposed hybrid scheme can maintain the same quality of secondary SP-frames with lower bitrates. For simplicity but without loss of generality, the following simulations are focused on the bit-counts of secondary SP-frames when different schemes are adopted. Figs. 8 and 9(a) (f) show a frame-by-frame comparison of size reduction in secondary SP-frames for different sequences in both switching directions. Fig. 8(a) (f) demonstrates high-to-low bit rate switching (Q p ¼ 8) while Fig. 9(a) (f) depicts low-to-high bit rate switching (Q p ¼8 ). In these figures, the values of the Y-axis mean the average size reduction of secondary SP-frames in percentage of our proposed hybrid scheme and the QDC- ME scheme over the pixel-me scheme. he positive value indicates the tested schemes generate less bit-count as compared with pixel-me whereas the negative value indicates that the tested schemes require more bit-count as compared with the pixel-me. In the high-to-low bit rate switching (switching down) scenario as shown in Fig. 8, the proposed hybrid scheme can substantially reduce the size of secondary SP-frames, about 7.%,.%, %,.%,.% and.% in average and up to.%, 7%,.%, %, 7% and % in Riverbed, Duckstakeoff, Shuttlestart, Crew, Mobisode and Mobisode, respectively, comparing to the traditional pixel-me scheme. Similarly, in the low-to-high bit rate switching (switching up) case as illustrated in Fig. 9, size reduction of secondary SP-frames using the hybrid scheme is also very significant, about.%,.%,.%,.9%,.7% and.7% in average and up to.%,.%, 8%, %, % and % in Riverbed, Duckstakeoff, Shuttlestart, Crew, Mobisode and Mobisode, respectively. From Figs. 8 and 9, it can be seen that our hybrid scheme can outperform in all secondary SP-frames for all sequences even though can also have remarkable size reduction over the pixel- ME. he significant improvement of the proposed scheme is due to the flexibility of performing motion estimation and compensation in both pixel and QDC domains. he process of selecting an appropriate domain for motion estimation can prevent the case when the MB has similar pixel values in which the motion estimation operated in able Average size of secondary SP-frames with different Q p for the switching-down scenario. he numbers in brackets represent the savings for various schemes as compared with pixel-me. Q p 8 pixel-me [] pixel-me [] (KBits) pixel-me [] Riverbed (.7%) (.8%) (.9%) ( 7.9%) (.%) (.%) Duckstakeoff (.8%) (.89%) (.%) (.%) (.%) (.%) Shuttlestart (.79%) (.%) (.98%) (.9%) (.97%) (.9%) Crew (.99%) (.888%) (.8%) ( 8.9%) (.%) (.8%) Mobisode 8 9 (.9%) (.9%) (.%) (.%) (.8%) (.7%) Mobisode (.9%) (.88%) (.%) (.9%) (.%) (.7%) able Average size of secondary SP-frames with different Q p for the switching-up scenario. he numbers in brackets represent the savings for various schemes as compared with pixel-me. Q p 8 pixel-me [] pixel-me [] pixel-me [] Riverbed (.9%) (.89%) (.8%) (.%) (.%) (.%) Duckstakeoff (.788%) (.7%) (.8%) (.99%) (.9%) (.%) Shuttlestart (.99%) (.%) (.87%) (.%) (.%) (.7%) Crew (.8%) (.87%) (.%) (.%) (.997%) ( 8.%) Mobisode (.9%) (.99%) (.8%) (.79%) (.88%) (.8%) Mobisode (.%) (.7%) (.%) (.9%) (.89%) ( 8.%)

12 K.-K. Lai et al. / Signal Processing: Image Communication 7 () pixel domain generates fewer bits than, as explained in Section.. he evidence of this phenomenon is also depicted in Figs. 8 and 9 where a number of secondary SP-frames of the test sequences generated by introduce more bits in comparison with pixel- ME. he proposed hybrid algorithm effectively combines two existing techniques at MB level during the encoding of each SP-frame. he combination is facilitated by the flexible explicit mode in FMO, and is then controlled by the bitcount from the corresponding MB of the primary SP-frame. By making use of this arrangement, the overwhelming reduction in Figs. 8 and 9 of the proposed hybrid scheme highlights the importance of using a proper domain at MB level for motion estimation in secondary SP-frame coding. By the same token, ables and show the average size of secondary SP-frames with different Q p for switching-down and switching-up scenarios, respectively. Besides, Figs. and (a) (f) show the average percentage reduction in size of secondary SP-frames. Note that the quantization parameters were varied from to with a step-size of and the switching processes were from and to Q p ¼. ables and illustrate that even though the average size of secondary SP-frames using is smaller than that using pixel-me, the secondary SP-frame encoded by the proposed hybrid scheme requires the smallest bit-counts. It is obvious from Fig. (a) (f) and Fig. (a) (f) that using the proposed hybrid scheme, the size of secondary SP-frames can be remarkably reduced for various quantization parameters. Moreover, it is interesting to take a closer look at a frame of Shuttlestart, as depicted in Fig.. his frame contains a space shuttle launching with large amount of smoke. In this sequence, it includes scenes with challenging motion, which is suitable for. On the other hand, the MBs located at the sky, as highlighted in Fig., have inferior performance for, as the pixels in this area have similar values. he results in Figs. and show that the hybrid scheme can make good use of pixel-me and Fig.. Size reduction of secondary SP-frames in percentage achieved by the proposed hybrid scheme and over pixel-me [] with different for the switching-down scenario in (a) Riverbed, (b) Duckstakeoff, (c) Shuttlestart, (d) Crew, (e) Mobisode, and (f) Mobisode.

13 K.-K. Lai et al. / Signal Processing: Image Communication 7 () Fig.. Size reduction of secondary SP-frames in percentage achieved by FMO and over pixel-me [] with different Q p for the switchingup scenario in (a) Riverbed, (b) Duckstakeoff, (c) Shuttlestart, (d) Crew, (e) Mobisode, and (f) Mobisode. in ables and for switching-down and switching-up scenarios, respectively. All the simulations were carried out on a PC with an Intel Core M Quad Q9 CPU at. GHz and GB memory. We also demonstrate the savings of the proposed hybrid scheme over the scheme in ables and. Dime in these tables represents the percentage change of the average encoding time of the hybrid scheme over and it is calculated as follows: Dime ð%þ¼ ime ime QDC ME ime QDC ME ðþ Pixel-ME Fig.. Different areas using and Pixel-ME []. o compare the computational complexity required by various schemes, the average encoding time of secondary SP-frames with different Q p was measured and tabulated where ime and ime denote the encoding time used by the scheme and the proposed hybrid scheme, respectively. Owing to selecting an appropriate domain for motion estimation in secondary SP-frame coding, not all the blocks need to be encoded using QDC- ME. As a result, it can be easily seen that the proposed hybrid scheme can substantially reduce the computational

14 K.-K. Lai et al. / Signal Processing: Image Communication 7 () able Average time usage of secondary SP-frames with different Q p for the switching-down scenario. Q p 8 pixel-me [] (s) [7] (s) (s) Dime (%) pixel-me [] (s) [7] (s) (s) Dime (%) pixel-me [] (s) [7] (s) (s) Dime (%) Riverbed Duckstakeoff Shuttlestart Crew Mobisode Mobisode able Average time usage of secondary SP-frames with different Q p for the switching-up scenario. Q p 8 pixel-me [] (s) [7] (s) (s) Dime (%) pixel-me [] (s) [7] (s) (s) Dime (%) pixel-me [] (s) [7] (s) (s) Dime (%) Riverbed Duckstakeoff Shuttlestart Crew Mobisode Mobisode complexity of by %, %, %, %, % and % for the switching-down scenario and %, %, %, %, 7% and % for the switching-up scenario in Riverbed, Duckstakeoff, Shuttlestart, Crew, Mobisode and Mobisode, respectively, as shown in ables and. Not only encoding time can be saved, but our proposed hybrid scheme can also reduce greatly the bit-counts of secondary SP-frames.. Conclusion In this paper, an adaptive approach for motion estimation in coding H./AVC secondary SP-frames has been proposed. he proposed hybrid scheme combines motion estimation techniques, which are operated at two different domains: QDC domain and pixel domain. he QDCdomain motion estimation in secondary SP-frame coding offers higher coding efficiency for macroblocks with complex spatial or temporal activities, but it increases encoding time. he pixel-domain motion estimation is more suitable for macroblocks in homogeneous area, and requires less encoding time. A new measurement of inter-frame correlation based on the bit-counts of the macroblocks in primary SP-frames has been proposed to determine the combination of the two techniques. he hybrid scheme adaptively selects QDC or pixel domain motion estimation at macroblock level, which has the advantages of utilizing both domains in coding secondary SP-frames. Owing to the adoption of motion vector prediction in H./AVC, unsystematically mixing motion vectors obtained from the two different domains might increase the bit-count required by a secondary SP-frame. By making use of an explicit mode in FMO, defining the patterns of the slice groups according to the bit-counts of primary SP-frames is another contribution of this paper. Experimental results showed that the proposed measure of determining the proper domain for motion estimation in secondary SP-frame coding is effective. he hybrid algorithm significantly reduces the bit-counts of secondary SP-frames. Additionally, the computational complexity can be saved tremendously comparing to the QDC scheme. With the proliferation of HDV and multimedia applications, a new video standard called High Efficiency Video Coding (HEVC) is currently under development for coding video from QVGA ( ) up to 8p and Ultra HDV (78 ) [9]. It aims at providing high coding gain with % bitrate savings over H./AVC at the same video quality representation, probably at the expense of increased computational complexity. It is expected that bitstream switching will still be an important issue in entertainment-quality video services. Consequently, seamless switching with SP-frames has already been adopted in the newest reference software. Our proposed technique could also be extended to HEVC with new coding tools, for instance, more choices of coding modes, larger DC size and, as expected, larger search window for ultra-hdv supported by HEVC. Computational reduction of is of great importance when SP-frames are used in HEVC. It is anticipated that the encoding complexity and the bit-counts of secondary SP-frames will be reduced with the help of the proposed technique. Acknowledgments he work described in this paper is partially supported by the Centre for Signal Processing, Department of Electronic and Information Engineering, he Hong Kong Polytechnic

15 K.-K. Lai et al. / Signal Processing: Image Communication 7 () University and a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (PolyU /8E). Ki-Kit Lai acknowledges the research studentshipsprovidedbytheuniversity. References [] R. Kurceren, M. Karczewicz, Synchronization-predictive coding for video compression: the SP frames design for JV/H.L, in: Proceedings of the IEEE International Conference on Image Processing, vol., September, USA, pp. 97. [] M. Karczewicz, R. Kurceren, he SP- and SI-frames design for H., IEEE ransactions on Circuits and Systems for Video echnology (7) () 7. [] C.P. Chang, C.W. Lin, R-D optimized quantization of H. SPframes for bitstream switching under storage constraints, in: Proceedings of the IEEE International Symposium on Circuits and Systems, vo., Kobe, Japan, May, pp.. [] E. Setton, Bernd Girod, Rate-distortion analysis and streaming of SP and SI frames, IEEE ransactions on Circuits and Systems for Video echnology () () 7 7. [] W.. an, B. Shen, Methods to improve coding efficiency of SP frames, in: Proceedings of the IEEE International Conference on Image Processing, October, Atlanta, USA, pp.. [] K.K. Lai, Y.L. Chan, C.H. Fu, W.C. Siu, A quantized transform-domain motion sstimation technique for H. secondary SP-frames, in: Proceedings of the Pacific Rim Conference on Multimedia, October 7, Hong Kong, pp [7] K.K. Lai, Y.L. Chan, W.C. Siu, Quantized transform-domain motion estimation for SP-Frame coding in viewpoint switching of multiview video, IEEE ransactions on Circuits and Systems for Video echnology () () 8. [8] K.K. Lai, Y.L. Chan, C.H. Fu, W.C. Siu, Viewpoint switching in multiview videos using SP-frames, in: Proceedings of the IEEE International Conference on Image Processing, October 8, San Diego, USA, pp [9] Joint Video eam of ISO/IEC MPEG and IU- VCEG, IU- Recommendation H. Advanced video coding for generic audiovisual services,. [] G. Sullivan,. Wiegand, Video compression from concepts to the H./AVC video coding standard, in: Proceedings of the IEEE 9 () () 8. [] D. Marpe,. Wiegand, G.J. Sullivan, he H./MPEG- advanced video coding standard and its applications, IEEE Communications Magazine (8) (). [] K. Suhring, H. Reference Software JM.,. Available: / [] J. Yeh, M. Vetterli, M. Khansari, Motion compensation of motion vectors, in: Proceedings of the IEEE International Conference on Image Processing, October 998, Washington DC, USA, pp [] G. Laroche, J. Jung, B. Pesquet-Popescu, RD optimized coding for motion vector predictor selection, IEEE ransactions on Circuits and Systems for Video echnology 8 (9) (8) 7 7. [] S. Wenger, H./AVC over IP, IEEE ransactions on Circuits and Systems for Video echnology (7) (). [] H. Chen, Z. Han, R. Hu, R. Ruan, Adaptive FMO selection strategy for error resilient H. coding, in: Proceedings of the International Conference on Audio, Language and Image Processing, July 8, Shanghai, China, pp [7] S.K. Im, A.J. Pearmain, Error resilient video coding with priority data classification using H. flexible macroblock ordering, IE Image Processing () (7) 97. [8] Recommended Simulation Common Conditions for Coding Efficiency Experiments Revision, IU- SG/Q VCEG Doc. VCEG- AJ, San Diego, USA, July 8. [9] WD: Working Draft of High-Efficiency Video Coding, Joint Collaborative eam on Video Coding (JC-VC) IU- SG WP and ISO/IEC JC/SC9/WG Doc. JCVC-D, Daegu, Korea, January.