University of Bristol - Explore Bristol Research. Link to published version (if available): /ICIP

Transcription

1 Anantrasirichai, N., & Canagarajah, C. N. (2010). Spatiotempora superresoution for ow bitrate H.264 video. In IEEE Internationa Conference on Image Processing. (pp ) /ICIP Link to pubished version (if avaiabe): /ICIP Link to pubication record in Expore Bristo Research PDF-document University of Bristo - Expore Bristo Research Genera rights This document is made avaiabe in accordance with pubisher poicies. Pease cite ony the pubished version using the reference above. Fu terms of use are avaiabe: Take down poicy Expore Bristo Research is a digita archive and the intention is that deposited content shoud not be removed. However, if you beieve that this version of the work breaches copyright aw pease contact open-access@bristo.ac.uk and incude the foowing information in your message: Your contact detais Bibiographic detais for the item, incuding a URL An outine of the nature of the compaint On receipt of your message the Open Access Team wi immediatey investigate your caim, make an initia judgement of the vaidity of the caim and, where appropriate, withdraw the item in question from pubic view.

2 SPATIOTEMPORAL SUPER-RESOLUTION FOR LOW BITRATE H.264 VIDEO N. Anantrasirichai, C.N. Canagarajah University of Bristo, UK ABSTRACT Super-resoution and frame interpoation enhance ow resoution ow-framerate videos. Such techniques are especiay important for imited bandwidth communications. This paper proposes a nove technique to up-scae videos compressed with H.264 at ow bit-rate both in spatia and tempora dimensions. A quantisation noise mode is used in the superresoution estimator, designed for ow bitrate video, and a weighting map for decreasing inaccuracy of motion estimation are proposed. Resuts show improvement both in rate-distortion and perceived image quaity. Index Terms Video enhancement, Interpoation, 1. INTRODUCTION Teecommunications in rura areas suffer from probems of poor communications. Therefore the soution which is increasingy being turned to is the use of wireess communication systems such as sateite and GSM based technoogies. The advantage is ceary that these are cost-effective to set up, but at the price of imiting the bandwidth avaiabe. One of the potentia uses for such teecommunication systems is for transmission of video, for appications such as remote medicine, agricutura advice and education. These appications require the video to be of high enough resoution and quaity e.g. attempting to diagnose a medica probem from a harshy compressed QCIF video is not possibe. Due to the massivey imited bandwidth, existing video soutions (such as H.264) of acceptabe quaity are not suitabe in this scenario. Moreover, many such rura areas of interest are in deveoping countries, which therefore mean that high resoution cameras are not avaiabe, removing the possibiity of using scaabe video coding and other associated techniques. Two possibe soutions to this probem are super-resoution and frame rate up conversion which both fit around an existing compression system. Super-resoution is a spatia upsamping technique empoyed at the decoder which attempts to better estimate the high-resoution (from the camera) source video using information from mutipe ow resoution frames. Frame rate conversion is an equivaent technique which works in the tempora domain: extra frames are interpoated between the ow frame rate decoded video in an attempt to smooth the motion. Super-resoution is an area which is of considerabe interest, however the majority of work uses a downsamped version of the high-resoution video source, possiby with added noise very itte work uses source video which has been compressed. Therefore athough the work has ots of vaid concepts, some of the modeing and assumptions (e.g. that any noise present is Gaussian) are i-formed. This paper proposes a nove technique for upsamping ow bitrate video compressed using H.264, which uses information which is present in an H.264 bitstream to perform spatiotempora super-resoution. We aso introduce a quantisation noise mode used in the super-resoution estimator, particuary for ow bitrate video, and a weighting map for decreasing inaccuracy of motion estimation. The remainder of this paper is organised as foows: section 2 discusses reated work in both super-resoution estimation and frame rate conversion, before the proposed methodoogy is presented in section 3. Resuts are reported in section 4 and the paper is concuded and further work suggested in section RELATED WORK As mentioned in the introduction, itte work has been carried out on the appication of super-resoution to compressed video. However a ot of the background to super-resoution is of interest. Research on super-resoution can be divided into two main streams spatia and tempora. Section 2.1 introduces reated work on the former whist section 2.2 discusses the atter Spatia Super-Resoution Estimation Super-resoution (SR) image reconstruction increases the resoution of an image by observing mutipe ow-resoution (LR) images. In video sequences, successive frames are empoyed to construct a high-resoution (HR) frame. Athough the basic concept of the SR agorithm is simpe, there are many probems reated to perceptua quaity and restriction of avaiabe data. The LR images are typicay aiased and have sub-pixe shifts between one another, and the different data are possiby ocated at the same point in the HR image and some points in the HR image do not correspond to any information from the LR images. The first proposed SR technique was formuated in the frequency domain due to simpicity and cost [1], but exhibits high sensitivity to mode errors. Later the spatia domain has

3 been considered but the eary approaches are generay compicated. Consequenty maximum ikeihood estimation has been introduced with proper prior information and the assumpsion of additive Gaussian noise [2]. For the reconstruction process, Bayesian interpretation and the Tikhonov reguarisation is empoyed as statistica estimation probem. The fundamenta SR techniques have been reviewed in [3]. The newest technique deveoped in [7] introduces a spatiotempora video super-resoution using a mutidimensiona kerne regression. Most existing SR techniques have been proposed for uncompressed images. Unfortunatey, super-resoution agorithms designed for uncompressed data do not perform we when directy appied to decompressed image sequences, especiay for high compression rates. The reason is that the quantization error introduced during the compression process is often the dominant source of error when the compression rate is high and this error is not modeed. In [4], a method for SR MPEG video is introduced using quantisation information embedded in the compressed video stream. A Bayesian SR reconstruction technique is used to mode compression and expoit the quantization step size information for reconstruction [5]. This technique shows promising resuts for high bitrate; however, the image quaity is insignificanty improved at ow bitrate (256kbps). The method appied to compressed video has been further deveoped by improving the registration precision and the prior mode [6] Frame Rate Up-Conversion Decreasing frame rate generay saves bits transmitted/stored but produces artefacts of tempora visua degradation. Frame rate up-conversion (FRUC) is then required at the receiver. Motion compensated frame interpoation (MCFI) is the most common technique for interpoating frame as it provided simpicity with desirabe performance. The concept uses the motion vectors estimating motion between the existing frames, normay forward and backward frames, to construct the interva frame. The interpoated frame ocated between these two frames consequenty uses motion directions with fractiona vaues depending on tempora distance between the interpoated frame and two reference frames. This simpe approach cannot dea with occusions where hoes are present in the interpoated frame. Authors in [8] proposed the bi-directiona motion estimation by dividing the interpoated frame into nonoverapping bocks. Then the motion vector for each bock is obtained by positioning the forward and backward reference frames in opposite directions. Later, they proposed a variabe bock size to achieve better performance [9]. 3. PROPOSED SCHEME The proposed scheme enhances a decoded LR ow-framerate video by appying SR method to construct a HR ow-framerate video, whist appying a FRUC method to construct a LR highframerate video in parae. Afterward the FRUC method enhances the HR ow-framerate video using the prepared LR high-framerate video to generate a HR high-framerate video. We appy spatia super-resoution before frame interpoation (SR FI) since it processes faster than the aternative order (FI SR) as the more compex SR is working with fewer frames. Generay appying FI SR yieds better quaity compared to SR FI because the iterations of the SR process (wi be expained in 3.1) is abe to correct some of the errors produced from the FI process. However, in our scheme the FI for the HR video expoits the interpoated frame of the LR video as a guide thereby achieving comparabe quaity (as shown in the resut section) Spatia super-resoution The decoded ow-resoution frame y k (formed by exicographica ordering) is reconstructed using motion vectors v k,i and reference frames[ y i ( as shown in equation 1 [5]. y k = T 1 Q T g k )] C(v k,i )y i + C(v k,i )y i (1) i i where C( ) and Q[ ] represent the prediction and quantisation processes, respectivey, and T and T 1 represent the forward and inverse transform matrix, respectivey. The LR image g k reates to its HR version f k as g k = AHf k + n k, where H is a bur fiter, A is a decimation operation and n k is acquisition noise. The reationship between y k and f k is irreversibe. The estimator is therefore empoyed to estimate f k. Here we assume that the quantisation noise is dominant and the compression parameters can be extracted from bitstream. A Bayesian approach is consequenty used as a posteriori probabiity density function (PDF) of f k can be estabished. The estimated HR frame ˆf k is then found from a maximum a-posteriori (MAP) estimator. { ˆf k = arg max og } p(y f k ) + og p(f k ) (2) f k { p(y f k ) exp 1 2 (y AHC(d,k )f k ) T } (3) (y AHC(d,k )f k ) p(f k ) exp { λ 1 f k } (4) where K 1 is an inverse covariance matrix and d,k is a dispacement in the HR frames corresponding to the motion vectors v,k in equation 1. Ringing artefacts introduced by the coarse quantisation is used for modeing the first density function p(f k ). Note that H.264 has debocking fiter, so the bocking artefact is not incuded in p(f k ). The f k can be seen as the reguarization term defined as f k = (f xx fy 2 2f x f y f xy + f yy fy 2 )/(fx 2 + fy 2 ) [6], where λ 1 is a weighting constant (Here λ 1 = 0.1). Then equation 2 can be rewritten as k+t F ˆf k = arg min (y f k AHC(d,k )f k ) T =k T B (5) } (y AHC(d,k )f k ) + λ 1 f k In the foowing sections the proposed quantisation noise mode for ow bitrate video is described before the technique

4 for using the motion estimation error to decide how much the dispacement can be trusted is outined Quantisation Noise Mode The previous work on spatia super-resoution for H.264 compressed video has been primariy concerned with high bit-rate scenarios and as such has modeed the quantisation noise as uniform for each frequency. This assumption does not hod for ow bit-rate video, where the quantisation noise is distributed in a Lapacian fashion. The variance of these distributions varies across different frequency bands, with higher frequencies yieding ower variance. The covariance in frequency domain K T for each 4 4 bock transform is found here experimentay by ooking at five standard test sequences (Foreman, Mobie, Akiyo, Bus & Carphone), and is shown in equation 6. K T = q where q is a quantisation step size embeded in the decoded bitstream. Then the covariance matrix in equation 3 wi be K = T 1 K T T 1T, where K T is a diagona covariance matrix of repeat repositioned K T associating to y Motion Estimation Error The prediction process for the dispacement C(d,k ), shown in equation 5, empoys bock-based matching with initia motion vectors extracted from H.264 video. That is, d,k = m v,k, where m is the proportion between the size of the HR frame and the LR frame. Subsequenty refining dispacement is operated in a sma search window. To reduce the error incurred for instances of dispacement of homogeneous bocks, the grid of bocks is offset on even iterations by haf a bock. The macrobock mode and decoded residua from the H.264 bitstream are used to impy the areas of y which are difficut to match with areas in the reference frames. That is, the areas with high energy ɛ r decoded residua, or encoded with intra mode are ikey to be occusions. Simiary the areas with high energy of the compensated error ɛ c = f C(d,k )f k coud be occusions. ɛ r and ɛc are empoyed to create the weighting maps to be used in equation 5 as foows: W x = 1 λ 2 (ɛ x )2 1 + λ 2 (ɛ x, x {r, c} (7) )2 1 where λ 2 is a decreasing rate. Here λ 2 is chosen as 100 so is defined as 0.5 when the intensity error is 10. Using the steepest descent agorithm, the minimisation of equation 5 can be found as W x q 10 (6) f n+1 k = fk n α k+t F =k T B W c C(d k,)h T A T W r ( y AHC(d,k )fk n ) ] + λ 1 f k where f n+1 k and fk n are the estimated HR frame at the (n+1)th and nth interations, α is a reaxation parameter controing rate of convergence (Here α = 1). The same maps are used to generate the initia vaue, fk 0, i.e. using the weighted average of the adjacent frames as shown in equation 9 and the Wk c is 1. Note that if ˆf does not yet exist, biinear interpoation is empoyed. f 0 k = k+t F =k T B W c C(d,k)ˆf k+t F i=k T B W c i 3.2. Frame Rate Up-Conversion H.264 is a motion compensated codec, which means there is tempora information avaiabe within the bitstream which can be used in the motion estimation process. In the proposed method the motion vectors and macrobock modes present in the H.264 bitstream are used as the initia parameters for biatera motion estimation. A frame to be interpoated is first divided into bocks each of which in turn is estimated as the average of bocks of pixes in the neighbouring frames y F, y B which produce the minimum weighted sum of absoute differences, cacuated as shown in equation 10 and 11: (8) (9) w d = λ 3 d λ 3 d 2 (10) SAD d = w d y F (i + n F v i + d i, j + n F v j (i,j) B +d j ) y B (i n B v i n B d i, j + n B v j n B d j ) n F n F (11) where the weight w d is used to promote motion vectors which are cose to the initia one. This prevents a distant bock with homogeneous detai from being seected as a match between frames. d = (d i, d j ) is a distance from the initia motion vectors v i, v j extracted from H.264 bitstream, n F and n B are the tempora distances to y F and y B, respectivey. λ 3 is a scaing parameter. Here λ 3 is chosen as 1 25 so w is defined as 0.5 when the distance is 5 pixes from the initia motion vector. Afterward an adaptivey weighted vector-median fiter is empoyed to suppress outiers [9]. The frame rate of the LR video is enhanced using proposed method expained above. The interpoated frames are then upsamped to high resoution using biinear interpoation. Subsequenty as the current frame the motion compensation is appied using the neighbouring HR frames pre-generated via SR method as references. Finay the motion is smoothed in the same way appied to LR frames.

5 4. RESULTS The proposed scheme was tested with three standard test sequences: Foreman, Mobie, Carphone. The LR video sequences (QCIF format) were coded with H.264 at 15 frames per second (fps) and were enhanced to CIF format at 30 fps. The resuts were compared with the fu resoution videos coded with H.264 and the existing agorithms introduced in [5] and [7]. Figures 1 and 2 show the resuts of the proposed technique for three different sequences compared to the other agorithms discussed. It can be seen that at ow bitrate (50kbps) the proposed scheme exhibits a 30kbps (37.5%) saving over the fu resoution version compressed with H.264 whist maintaining the same PSNR. The proposed scheme aso outperforms the scheme proposed in [5], which empoys a uniform distribution for modeing quantisation noise, by upto approximatey 0.5 db consistenty across a bitrates. The agorithm proposed in [7] is incuded in the experiment in order to demonstrate the fact that agorithms which obtain promising resuts for uncompressed videos don t necessariy perform we with heaviy compressed data. For Mobie sequence, the proposed scheme achieves ess impressive improvement in terms of PSNR over other methods compared the the other sequences (figures 1 & 2). This is due to the high eve of detai it contains, which is much more senstive to the the downsamping stage present at the encoder and hence makes recovery far more difficut. Note that using PSNR to evauate the perforamce of FI is possiby not suitabe because the reason interva frames are inserted is to produce tempora smoothness, to increase the subjective quaity. The agorithm isn t attempting to estimate the origina frames so much as it is attempting to make the video ess irritating for the observer. PSNR (db) H.264 Proposed FI->SR[Sega] 3DKerne[Takeda] bitrate Fig. 1. Rate-distortion performance of Foreman 5. CONCLUSIONS & FUTURE WORK PSNR (db) H.264 Proposed FI->SR[Sega] 3DKerne[Takeda] bitrate Fig. 2. Rate-distortion performance of Carphone tia and tempora dimensions. A quantisation noise mode used in the SR estimator for ow bitrate video is proposed. Aso the accuracy of the motion estimation is weighted to avoid error accumuation produced in the iterative process. The proposed scheme achieves a bitrate reduction compared to transmitting HR video and gains an improvement of image quaity compared to the existing schemes. The more accurate quantisation noise mode wi be considered using the detais of each video in the future. 6. REFERENCES [1] T Huang and R Tsai, Muti-frame image restoration and registration, Adv. Comput. Vis. Image Process, vo. 1, pp , [2] M Ead and Y He-Or, A fast super-resoution reconstruction agorithm for pure transationa motion and common space-invariant bur, IEEE Trans. Image Processing, vo. 10, no. 8, pp , Aug [3] S. C Park, M. K Park, and M. G Kang, Super-resoution image reconstruction: a technica overview, IEEE Signa Processing Mag., vo. 20, no. 3, pp , May [4] Y Atunbasak, A Patti, and R Mersereau, Super-resoution sti and video reconstruction from mpeg-coded video, IEEE Trans. Circuits Syst. Video Techno., vo. 12, no. 4, pp , Apr [5] C Sega, A Katsaggeos, R Moina, and J Mateos, Bayesian resoution enhancement of compressed video, IEEE Trans. Image Processing, vo. 13, no. 7, pp , Juy [6] C Wang, P Xue, and W Lin, Improved super-resoution reconstruction from video, IEEE Trans. Circuits Syst. Video Techno., vo. 16, no. 11, pp , Nov [7] H Takeda, P Mianfar, M Protter, and M Ead, Superresoution without expicit subpixe motion estimation, IEEE Trans. Image Processing, vo. 18, no. 9, Sep [8] B.-T Choi, S.-H Lee, and S.-J Ko, New frame rate up-conversion using bi-directiona motion estimation, IEEE Trans. Consumer Eectron., vo. 46, no. 3, pp , Aug [9] B.-D Choi, J.-W Han, C.-S Kim, and S.-J Ko, Motion-compensated frame interpoation using biatera motion estimation and adaptive overapped bock motion compensation, IEEE Trans. Circuits Syst. Video Techno., vo. 17, no. 4, pp , Apri This paper presents a nove technique for upsamping ow bitrate video compressed using H.264. The coding parameters embeded in the bitstreams are used to up-scaing both in spa-