Classification Based Mode Decisions for Video over Networks

Transcription

1 Classfcaton Based Mode Decsons for Vdeo over Networks Deepak S. Turaga and Tsuhan Chen Advanced Multmeda Processng Lab Tranng data for Inter-Intra Decson Inter-Intra Decson Regons pdf Energy 4 3 Energy 4 3 Inter 2 2 Feature vector x mrmad Intra mrmad Techncal Report AMP - December 2 Electrcal and Computer Engneerng Carnege Mellon Unversty Pttsburgh, PA 523

2

3 Abstract A vdeo encoder has to make many mode decsons n order to acheve the goal of a low bt rate, hgh qualty, and fast mplementaton. We propose a general classfcaton based approach to makng such mode decsons accurately and effcently. We frst llustrate the approach usng the Intra-Inter codng mode decson. We then focus on the decson to skp or code a frame for rate control of vdeo over networks. Usng the classfcaton based approach we show mprovement n the rate-dstorton sense. We then extend the work to scalable vdeo codng n choosng between scalablty modes and examne the performance of our approach over error prone networks, usng smulated packet losses. I. Introducton A vdeo encoder converts raw vdeo data nto a standard specfed btstream, whch s then transmtted over the network and reconstructed to vdeo by the decoder. Ths converson from vdeo to btstream s done to acheve compresson wthout sacrfcng on the qualty of the reconstructed vdeo. Real tme vdeo encodng requres fast mplementaton of the codng process. Hence vdeo encoders have to perform well n terms of the speed-qualty-bt rate tradeoff. Inherent n the codng process are many mode decsons that mprove one or more of the parameters n ths speed-qualty-bt rate tradeoff. Vdeo standards such as MPEG [] and H.263 [2] specfy the btstream syntax completely, but allow for some optmzatons n the encodng process n terms of algorthms and mode decsons used. These optmzatons are allowed as encoders desgned for dfferent applcatons need to be optmzed dfferently. For nstance when the applcaton requres real-tme encodng, the speed of the codng becomes crtcal, whle n applcatons such as off-lne codng, the qualty and the codng effcency are more crtcal than the speed. In ths paper we show that many mode decsons n the codng process can be consdered as bnary hypothess testng problems that are well understood n tradtonal classfcaton theory. The goal of a mode decson s to mnmze a cost that may be defned n terms of one or more of the parameters of the speed-qualty-bt rate tradeoff. The optmal mode decson s typcally data dependent. In theory, for each mode decson, we can try all the possble modes, evaluate the cost correspondng to each mode, and choose the one wth the smallest cost. However, such an Contact Author: Prof. Tsuhan Chen, Electrcal and Computer Engneerng, Carnege Mellon Unversty, 5 Forbes Avenue, Pttsburgh, PA 523. Emal: tsuhan@cmu.edu Accepted for publcaton n IEEE Transactons Multmeda, Specal Issue on Multmeda over IP, March 2.

4 exhaustve search approach s mpractcal due to ts complexty. An alternatve s to dentfy features that can be easly computed from the vdeo data, and are good ndcators of whch mode would be optmal. In order to do so, we frst collect vdeo data and use exhaustve search to label the data wth the optmal mode decsons. We then estmate probablty densty functons for these features under dfferent hypotheses, correspondng to the dfferent optons n the mode decson. We then transform ths mnmzaton of cost to a more tradtonal error probablty mnmzaton problem and use standard classfcaton technques lke the lkelhood rato test to solve t. We evaluate the performance of ths classfcaton approach usng two mode decsons that are part of the vdeo codng process. We frst use the Inter-Intra mode decson as an example and then focus on the mode decson for skppng or codng a frame for rate control purposes. As an llustraton of our approach we use the Intra-Inter mode decson. We evaluate the performance of the decson proposed n the Test Model Near Term (TMN ) [3]. Usng a new feature, the mean removed mean absolute dfference (mrmad) and our classfcaton based approach we mprove ths decson. More dscusson about the mrmad can be found n [4]. The focus of ths paper s on the mode decson to skp or code a frame for rate control of vdeo that needs to be transmtted over the network. The problem of rate control has been extensvely studed n lterature. Rate control s essental when we need to constran the output of a varable bt rate (VBR) encoder to the bt rate provded by the network wthout sacrfcng too much on the qualty of the decoded vdeo. Ths target bt rate that we have to acheve may be constant or varable dependng on the nature of the network. Lakshman et. al. [5] classfy VBR vdeo nto unconstraned, shaped, constraned and feedback dependng on the desgn of buffers for rate control, or algorthms used by the encoder to change the codng parameters usng feedback nformaton from the network. Chen and Wong [6] and Cho and Park [7] try to solve the rate control problem usng buffer control strateges. As aganst ths approach, Hsu, Ortega and Rebman [8] propose algorthms for rate control usng both source rate control as well as channel rate control over asynchronous transfer mode (ATM) networks. Most of the work n these papers tres to control the rate by changng the spatal qualty usng the quantzaton step sze and consder changng the frame rate for rate control as the last opton, when we cannot acheve the target rate. Song, Km and Kuo [9] propose rate control algorthms for low bt rate unconstraned VBR that allow for a varable target rate provded by the network. The algorthms they propose try to control the spatal and temporal qualty smultaneously to mprove the perceved qualty. They use frame skppng or frame rate changes n order to mprove the perceved qualty and the prmary algorthm to control the bt rate s through the use of the quantzaton step sze. Work by Martns, Dng and Feg [] tres to acheve rate control by combnng the effect of codng and

5 skppng a frame nto a composte cost functon. They, however lack granularty n ther work, as the decson they make s not on a frame by frame bass. They make a decson for the future usng nformaton from codng the current frame. As an example, f after codng a frame they feel that they need to skp frames, they do so wthout examnng any of the followng frames. Most of the work above uses models to estmate the relatonshp between bts needed to code frames and the quantzaton step sze. Ths s useful to estmate the quantzaton step sze needed to code the frame gven a target rate. Some of these models are proposed by Chang and Zhang [] and by Dng and Lu [2]. Most of exstng rate control algorthms modfy the quantzaton step sze to control the rate of VBR encoders. Frame skppng to control the bt rate s used only as a last resort,.e., when a buffer overflow occurs or when the buffer approaches overflow. Such algorthms do not consder the loss of qualty that occurs by ths arbtrary skppng of frames. Ths problem s more severe at low bt rates when t s dffcult to acheve the target rate by changng the quantzaton step sze alone and more frames need to be skpped. We propose a scheme for rate control that decdes ntellgently between changng the quantzaton step sze and skppng the frame to acheve the desred target rate. We look at the sutablty of skppng or codng a certan frame n terms of the mpact t has on the rate as well as the mpact t has on the qualty before makng a decson. We try to maxmze perceved qualty whle achevng a fxed target rate. We mplement classfcaton based strateges to decde between codng and skppng a frame. We extend our work on rate control to scalable vdeo codng n order to evaluate the performance of our scheme under lossy network condtons. We create an enhancement layer correspondng to the base layer generated usng our rate control strateges and examne the reconstructed vdeo qualty over dfferent smulated packet loss scenaros. Effectvely, ths s equvalent to choosng between SNR scalablty and temporal scalablty adaptvely, dependng on the vdeo data. The paper s organzed as follows. Secton II ncludes the dscusson of the general classfcaton based approach to makng mode decsons usng the Intra-Inter mode decson as an example. Secton III contans the use of ths approach towards the rate control problem usng an nstantaneous mode decson. Secton IV ncludes the descrpton of the mode decson allowng for lookng ahead at one future frame to be coded. We then extend ths work on rate control to scalable vdeo codng and ths s descrbed n Secton V. We then conclude wth the analyss of our scheme and descrbe some future work and possble extensons.

6 II. Classfcaton Based Approach to Mode Decson In ths secton we show how to convert a mode decson nto a bnary hypothess-testng problem. Let co represent the cost for makng a decson D and c represent the cost for makng decson D for a partcular codng unt. These decsons D and D may nclude mode decsons that may be at dfferent levels of the codng process,.e., at dfferent codng unts. For nstance these could be at the macroblock level, wth one macroblock beng the codng unt, where D could be to decde to code the macroblock usng ntra codng whle D could be to code the macroblock usng nter codng. These could also be at the frame level, wth one frame beng the codng unt, where D and D represent whether to code or skp the frame. The costs c could nclude the number of bts needed to code a block or frame gven that we have made a partcular decson. The cost could also nclude the dstorton ntroduced n the decoded vdeo as well as the tme needed to encode the vdeo accordng to the decson. The goal of our strategy s to make the decson that mnmzes ths cost. So every tme we need to make a decson we want to make one that results n the smaller cost. Ths strategy may be summarzed as below. Choose D f c Choose D f c < c < c or c or c A. Transformng Mode Decson nto Classfcaton Problem In prncple, to make the optmal mode decson one can try all the modes and choose the mode that has the lowest cost. However, computng the actual costs c before makng a decson s very computatonally ntensve as ths nvolves tryng ether decson to determne the cost. In order to reduce computatonal burden for the decson scheme we would lke to dentfy features that provde a good estmate of the cost for a decson, but do not requre as much computaton to evaluate. We would, thus lke to dentfy features that let us estmate whch of the two followng hypotheses H or H s true where H H : c : c c c < > c c < > For each codng unt we dentfy K features that we group together n a feature vector T x = x ]. In the optmal scenaro we could fnd features that perfectly represent the [, x, x K cost needed for a decson. However, n most practcal applcatons such features are dffcult to fnd. In most practcal applcatons, the decson strategy thus becomes sub-optmal n terms of

7 mnmzng the cost, however we are wllng to settle for ths sub-optmalty as along wth ths comes the beneft of small computatonal requrements. We want to buld a classfer that would take as nput the feature vector x of each codng unt and come up wth the probablty that H or H s true, whch would then enable us to make a decson approprately. We can come up wth such a classfer by tranng t wth sample data. Suppose we collect a set of M codng unts wth correspondng feature vectors x and assocated wth each of these feature vectors s a cost dfference, x x M d c c = wth c and c beng the true costs for decsons D and D respectvely for the -th codng unt. Let P of these codng unts have d <,.e., correspondng to when H s true and Q of these codng unts have d >, correspondng to when H s true, wth P + Q = M. For purposes of llustraton, we show these two sets of tranng vectors for a onedmensonal feature vector n Fgure. H true for these P vectors H true for these Q vectors d = c c d = c c Feature Vector x Feature Vector x Fgure. Illustraton of feature vectors In Fgure, the x-axs for both the cases corresponds to the value of the feature vector x (we are usng a -D vector for llustraton) for each codng unt whle the y-axs corresponds to the magntude of the cost dfference d between the two mode decsons for that codng unt. Feature Vectors of codng unts for whch H s true are shown on the left and are represented wth trangles whle feature vectors of codng unts for whch H s true are shown on the rght and represented wth squares. The magntude dfference d, for any codng unt, corresponds to the addtonal cost that we need to pay f we make the wrong mode decson for that unt. For nstance f we choose

8 tomakeadecson D, nstead of the rght decson D, for one of the codng unts represented by trangles, we ncur a cost c nstead of ncurrng the smaller cost c and so pay an addtonal cost d c c =. We may put together the two plots from Fgure to obtan the entre tranng feature space and ths s shown n Fgure 2. d T Feature Vector x Regon R Regon R Fgure 2. Tranng feature space All the feature vectors correspondng to the M codng unts are shown on the same plot and we use trangles and squares as before to separate the data nto the two classes. We now want to partton ths feature space usng our classfer so that the total addtonal cost that we have to pay for msclassfcaton,.e., makng the wrong decson for any codng unt, s as small as possble. For nstance we may partton ths space nto two regons, R and R, usng the threshold T as shown n Fgure 2. We choose to make decson D for all codng unts wth feature vectors to the left of the threshold,.e., n R, and decson D for all codng unts wth feature vectors to the rght of the threshold,.e., n R. For such a case we pay the addtonal cost d for each of the msclassfed codng unts,.e., squares n R and trangles n R.InFgure2 we show these as dark trangles and squares, as opposed to the lghtly colored ones, for whch we make the rght decson. We would lke our mode decsons to result n as small a total cost as possble. In order to do ths we need to choose R and R so that we mnmze the total addtonal cost from makng wrong mode decsons. Ths knd of a problem of parttonng the feature space n order to mnmze the total cost s remnscent of tradtonal classfcaton theory, except that n tradtonal classfcaton theory the vertcal axs corresponds to the probablty denstes of the feature vectors under the dfferent hypotheses, whle n our problem ths corresponds to the addtonal

9 cost we have to pay when we make a wrong decson. Hence, n order to use tradtonal classfcaton technques to solve our problem, we need to modfy our problem to ft the classfcaton scenaro. Our problem of mnmzng the total addtonal cost may be mathematcally wrtten as follows. mn d +, d R R x R > x R d d < These two regons, R and R should together span the entre space, and have no common sub-regons. However, we mpose no constrants on the shapes of these regons as long as they satsfy ths mnmzaton requrement. These regons may consst of non-contguous subregons and the boundares between them may be arbtrarly shaped. Hence we formulate the problem of mnmzaton n terms of choosng the regons and not n terms of specfyng lnear boundares or thresholds separatng them. Let wrtten as follows. N = d and N = d < mn R, R N d d > x R > d. Our mnmzaton problem may be equvalently N d N d + + N N N + N N x R < d As mentoned before, we would lke to use standard classfcaton technques to solve the problem and dentfy these regons R and R. Such technques nvolve estmatng the probablty densty functon (pdf) of the feature vector under the dfferent hypotheses and then usng these pdfs to dentfy the best regons. However, the feature vectors n our problem are dfferent from the feature vectors typcally used n classfcaton problems. Ths s because they not only have a certan value, but also a heght (correspondng to the addtonal cost) assocated wth them. In the typcal classfcaton scenaro the vectors do not have ths addtonal assocated heght. Essentally, we need to convert these feature vectors wth the assocated heghts to vectors that do not have these addtonal heghts, but we need to do ths wthout losng the mportant nformaton that the heghts carry. We may do ths transformaton by replacng a vector wth heght d wth d vectors at that locaton. In general t s not necessary that all the heghts d are ntegers, however wthout loss of generalty we can scale them approprately to make them

10 ntegers. We can thus modfy our feature space and our modfed feature space would look as shown n Fgure 3. T Feature Vector x T Feature Vector x R R R R Fgure 3. Transformaton of feature space from old feature space (left) to new feature space (rght) From Fgure 3 we can see that each vector n the old feature space s replaced by multple vectors at that locaton, the number of new vectors beng equal to the heght assocated wth the orgnal vector. Now, standard classfcaton technques may be appled n ths new feature space to estmate the pdfs for ths new set of vectors. We rewrte our mnmzaton problem n ths new feature space and then map t to the well-understood mnmum probablty of error classfcaton problem. We can thus use results from lterature to fnd the regons that mnmze our crteron n ths new feature space. Because of the way we transform the old feature space to the new feature space, the regons we fnd n ths new feature space are dentcal to the regons we desre n the orgnal feature space. Ths s because none of the tranng data ponts are dsplaced from ther orgnal postons. The detals of ths proposed scheme are presented n the followng paragraphs. In ths new feature space N s smply the total number of trangles and N s the total number of squares, where N and N are as defned before. We may defne two new hypotheses as followng. H : x H : x belongs to the trangle class belongs to thesquare class N Hence, n ths new feature space = P( H ), the probablty of a feature vector beng a N + N N trangle and smlarly = P( H ). N + N

11 d Also = P( x = x H ) when d <, because d s the number of trangles at x and N N d s the total number of trangles. Smlarly, = P( x = x H) when d >. Hence our N mnmzaton problem may be rewrtten n ths new doman as follows mnp( H = + ) P( x x H) P( H ) P( x = x R, R x R x R H ) In practce, nstead of usng these dscrete probabltes, we model the data usng a contnuous pdf consstng of a mxture of Gaussans. Ths s because n order to classfy a feature vector we need to have the probablty of occurrence of that vector. However we do not have these probabltes for new nput vectors that are not present n the tranng data set. By modelng the feature vector pdf usng a mxture of Gaussans we ensure that any feature vector may be classfed. These Gaussan mxtures are traned on the modfed feature vectors usng the Expectaton Maxmzaton (EM) algorthm, more detals on whch may be obtaned from [4]. An example of traned Gaussan mxtures n the modfed feature space s shown n Fgure 4. pdf Feature vector x Fgure 4. Modelng the data by a mxture of Gaussans The densty functon drawn wth the dotted lne corresponds to the data from the trangle class whle the densty functon drawn wth the sold lne corresponds to the data from the square class. Usng our contnuous pdfs we may rewrte our mnmzaton problem as follows. The functons p( ) mnp( H + ) p( x H) dx P( H ) p( x H ) dx R, R x R x R correspond to the contnuous pdf comprsng of a mxture of Gaussans. Ths knd of a mnmzaton problem s equvalent to a mnmum probablty of error classfcaton scheme n ths new feature space and t s well known from lterature [3] that the

12 regons may be determned usng the lkelhood rato test. The lkelhood rato test may be wrtten as p( x H ) > P( H ) p( x H ) < P( H ) H H Hence, n order to classfy a feature vector x, we look at the lkelhood rato for t and f ths exceeds the threshold obtaned from tranng, we beleve H s true and make decson D otherwse we beleve H s true and make decson D. In summary, ths lkelhood raton test defnes the decson regons n ths new space. As we mentoned before, the regons that we determne n ths new feature space are dentcally the regons that we desre n our orgnal feature space. So by transformng our feature space and mappng the problem to a wellunderstood mnmzaton problem we can obtan the soluton we desre. The entre classfcaton scheme may be summarzed as follows. Gven the tranng data and the cost dfferences, we frst transform the feature space to the new feature space and then estmate the apror probabltes, P H ) and P H ), as well as the class condtonal ( probablty densty functons, p x H ) and p x H ), usng the EM algorthm to tran the ( ( ( Gaussan mxture. Once we have these pdfs, we use the lkelhood rato test for a new nput feature vector correspondng to a codng unt and determne whch of the two hypotheses s more lkely to be true and usng ths result we make decson D or D for that codng unt. B. An Example: Intra versus Inter Mode Decson Ths decson s made for every macroblock (a 6 6 regon n a frame) n the vdeo sequence. Intra codng nvolves codng usng transform codng followed by quantzaton and entropy codng. As opposed to ths, Inter codng nvolves buldng a predcton for the current macroblock usng data from the prevous frame usng moton estmaton and compensaton and codng the resdue usng transform codng, quantzaton and entropy codng. For most macroblocks, Inter codng s more effcent n terms of compresson, however ths s not always true. When there s a scene change or when we have a hgh moton sequence the predcton for the macroblock from the prevous frame s lkely to be poor and n such cases t may be more effcent to use Intra codng as opposed to Inter codng. Hence the encoder has to decde between these for every macroblock. The decson that requres fewer bts s preferred. As convertng to bts s computatonally expensve, encoders use features as estmates for the bts. Typcally the energy (wth DC value removed) n the block s used as an estmate of the bts needed for ntra codng, whle the MAD s used as an estmate for the bts needed for nter

13 codng. For example, the mode decson as recommended by the TMN- of the H.263 standard s Intra Codng f MAD E x > T Inter Codng otherwse 6 6 For a 6 6 block E x = x, j mx s the energy, mx s the mean or the DC value of 256 = j= the block and T s an emprcally found threshold, specfed n the TMN as 5. We also test another feature, the mean removed MAD (mrmad) as a feature to estmate the bts needed for nter codng. Ths feature s very smlar to the MAD, except that nstead of takng the absolute pxel dfference and summng them up we frst remove the means from the blocks and then sum up the absolute pxel dfference. In order to collect tranng data we perform the exhaustve test for a varety of sequences and generate a sequence of values for the features and a sequence of the optmal decsons. These exhaustve tests nvolve actually computng bts needed for Intra and Inter codng and makng the rght decson,.e. the decson resultng n fewer bts. After collectng the sequence of rght decsons and the values of the features we correlate these wth the decson sequence. From experments we see that the energy s very well correlated wth the bts needed for ntra codng (correlaton coeffcent of.87). The MAD and the mrmad are features representatve of the bts needed for Inter codng. They have correlaton coeffcents of.9 and.94 respectvely wth bts for nter codng, ndependent of sequence. In order to test the sutablty of usng these features, we correlate these wth the optmal decson sequence (one determned usng the exhaustve test). The decson sequence s vewed as a sequence of +s and s wth + correspondng to Intra and correspondng to Inter. More detals on our computaton of correlaton coeffcents are ncluded n the Appendx. The correlaton coeffcents for each feature are presented n Table. Table. Correlaton coeffcents of features wth decson sequence Feature Correlaton coeffcent wth decson sequence Energy.322 MAD.474 mrmad.5 From the table we see that the mrmad s better correlated wth the decson sequence than the MAD, hence we choose ths as a feature. Although the MAD has a hgher correlaton wth the decson sequence than the energy, t s hghly correlated wth the mrmad. As the energy s representatve of the bts needed for ntra codng, we use ths feature, nstead of the MAD, along wth the mrmad for our classfcaton scheme.

14 We collect feature vectors from the Foreman, Coastguard and Slent sequences. Snapshots from these sequences are shown n Fgure 5. Fgure 5. Snapshots from Foreman (left) Coastguard (center) and Slent (rght) The sequences are n (QCIF) format at a frame rate of 3 Hz. We collected 792 feature vectors of whch we used 5 to tran our classfer pdfs. The number of tranng vectors s small as compared to the test set, but usng a larger tranng set does not mprove the performance of our classfer sgnfcantly. Usng the tranng data, we tran the Gaussan mxtures usng to obtan decson regons as shown n Fgure 6. Tranng data for Inter-Intra Decson Inter-Intra Decson Regons Energy 4 3 Energy 4 3 Inter mrmad Intra mrmad Fgure 6. Inter-Intra tranng data (left) and decson regons (rght) The plot n Fgure 6 shows tranng feature vectors on the left wth trangles correspondng to vectors that belong to the Intra class and squares correspondng to Inter class. On the rght we also show the decson regons that we obtan after tranng Gaussan mxtures on ths tranng data, wth the Intra decson regon enclosed nsde the black boundary. We can see that the decson regons are representatve of the tranng data and may consst of dsjont subregons, as we have for the Intra case. From the tranng data we can see that a lnear decson

15 boundary s not sutable n ths case. Thus, mposng no constrant on the shape of the decson boundary ads our classfer n fndng a better decson boundary. We acheved 98.2% correct classfcaton usng our mnmum probablty of error classfer. The classfcaton result shows a varaton of less than % across these dfferent sequences. The classfcaton scheme proposed n the TMN acheves around 92% correct classfcaton, because t uses a lnear decson boundary. Hence we can mprove the mode decson usng ths scheme. We then mplemented ths mode decson n the H.263 framework and observed that the correspondng savngs n total bt rate over the TMN decson (ncludng resdue bts, moton vector bts and overhead bts) were around 4.5~4.8% for dfferent vdeo sequences. III. Mode Decson for Skppng or Codng Frames: Instantaneous case The goal of ths mode decson s to decde between skppng and codng a frame n order to maxmze the perceved qualty whle achevng a target bt rate. In order to collect tranng data for our classfer, we frst mplement an exhaustve search based mode decson. We compute the effect of skppng as well as codng a frame on the qualty q and the bt rate r before makng a decson. In order to measure the perceved qualty we use a spato-temporal qualty metrc ntroduced by Wolf and Pnson [5]. In order to control the rate of coded frames, we change the quantzaton step sze usng the nverse quadratc model to relate the bt rate wth the quantzaton step sze. Ths model was proposed n [] and t relates the rate ( r ) to the quantzaton step sze a b (Q) usng r = +,wherea and b are constants that may be estmated usng tranng data. 2 Q Q We found that such a smple model cannot capture the varaton of bt rate wth quantzaton step sze across many dfferent scenaros, so we tran separate models,.e., parameters a and b, for low moton, medum moton and hgh moton sequences. More detals about the qualty metrc can be found n the appendx. Our cost s defned as a combnaton of the qualty and rate, defned as q + λr, where the factor λ s adjusted dependng on applcaton and we dscuss ths n more detal later. We compare ths cost for the skppng or codng and choose the one that requres the smaller cost. Smultaneously we collect the cost dfference and also evaluate some features that we use for tranng our classfer. Usng the method descrbed n Secton II, we tran the densty functons for our features and mplement our classfcaton based scheme.

16 We frst descrbe the exhaustve search based mode decson, after whch we descrbe the features that we choose for the classfer and fnally we nclude the results of our mplementaton. In all of the followng dscusson, the vdeo sequence s represented by a sequence of frames X ( n ), X ( n), X ( n + ) wth n representng the tme ndex. Snce we are usng lossy compresson technques the sequence of decoded frames may be represented as X ˆ ( n ), Xˆ ( n), Xˆ ( n + ). These may not be dentcal to the orgnal vdeo sequence. The prevous decoded frame s used as reference to code the current frame and when a frame s skpped t s replaced by the prevous decoded frame. The steps for the exhaustve search based mode decson are as follows. We frst compute the rate and qualty when we skp a frame. We replcate the prevous decoded frame to smulate skppng the current frame. We then estmate the qualty q of ths sequence of two frames{ X ˆ ( n ), Xˆ ( n) = Xˆ ( n ) }. The bt rate r s estmated by averagng the bts needed to code the past ten frames, settng the bts for the current frame to zero and multplyng by the frame rate. We estmate the bt rate usng a ten frame wndow as ths smooths out the fluctuaton due to large or small number of bts to code the current frame. We then estmate qualty and bt rate for codng the frame. We determne the bts avalable to code the current frame usng hstory nformaton and the target rate. We then estmate the quantzaton step sze needed to code ths frame usng the nverse quadratc model. Usng the prevous decoded frame as reference and the computed quantzaton step sze, we code the current frame and reconstruct t. We compute the qualty q2 of ths two frames sequence { ˆ ( n ), X ˆ ( n) } X and the bt rate r 2 as before. We then compare q + λr wth q2 + λr2 and decde whch of the two s better. The factor λ can be specfed by the user n terms of the relatve mportance of ether the rate or the qualty. In our tests we place a greater emphass on the qualty of the sequence. Ths s done by adjustng λ so that λ tmes the target rate s.5 that s comparable to the range of the qualty [-, ]. Ths s acceptable as we already use the quadratc model to try to control the bt rate. Ths decson strategy may be represented pctorally as n Fgure 7.

17 q Rate r Xˆ ( n ) skp Xˆ ( n ) code Rate r 2 X ˆ ( n ) q 2 Tme ndex n Fgure 7. Pctoral representaton of Exhaustve scheme Ths exhaustve scheme s very smlar to the Vterb decodng scheme [6] wth no lookahead allowed, as at every nstant n tme n, costs for both the paths (skppng the frame or codng t) are compared and the best path s chosen, wth the other beng dscarded. In fact ths rate control strategy can be extended to allow for look ahead and ths s descrbed n Secton IV. There we also nclude a greater dscusson relatng our scheme to the Vterb decodng scheme. Durng ths exhaustve mode decson we also evaluate some features. We start wth a large set of features and look at the correlaton of these features wth the decson sequence of the exhaustve search based mode decson to dentfy the features we use for our classfer. Some of the ntal features that we dentfed are descrbed n the followng. ) Sze of moton vectors. Ths s computed as the sum of the square length of all the moton vectors n the frame. 2) MAD or SAD as a measure of qualty of moton compensaton. Ths s obtaned as the sum of the MAD across all the macroblocks of the frame. 3) Measure of hgh frequency energy (HFE) n frame. Ths s obtaned by takng a frame, downsamplng t by a factor 2 horzontally and vertcally, then up-samplng t back to the orgnal sze and fndng the energy n the dfference between ths and the orgnal frame. Ths process maybevewedasnfgure8.

18 downsample upsample Compute energy HFE Fgure 8. Computaton of HFE In the above fgure down-samplng ncludes a pre-processng by a low pass flter and upsamplng ncludes a post-processng wth a low pass flter. 4) Bts avalable to code current frame. Ths may be obtaned from rate hstory. 5) Quantzaton step sze used for current frame. 6) Energy n the frame dfference between the current frame and prevous frame. We collected these features across dfferent sequences and across dfferent target rates usng our exhaustve search based mode decson. We then correlate these features wth the decson sequence that s vewed as a sequence of +s and s wth + correspondng to skppng a frame and correspondng to codng t. More detals of our computaton of these correlaton coeffcents are ncluded n the Appendx. The correlaton coeffcent for each of the features s ncluded n Table 2. Table 2. Correlaton coeffcents for features wth decson sequence Sequence Foreman Coastguard Slent Target (kbps) Sze of MV SAD HFE Avalable Bts Quant. Step Sze Frame dfference As can be seen from the table, most features are negatvely correlated wth the decson sequence whle the quantzaton step sze s postvely correlated. Ths s as expected because small moton vectors, small SAD, small HFE and a small frame dfference all mply that the current frame can be very well predcted by the prevous frame, thereby meanng that they are

19 smlar. Ths means that we can skp the frame, as we would then replace t wth the prevous frame. Ths bases the decson towards not codng the frame or a +. So a smaller value of each of these features corresponds to a large value n the decson sequence, hence a negatve correlaton coeffcent. A small number of avalable bts means that the qualty of codng the frame wll be poor, hence ths also tends to bas the decson towards skppng the frame, thereby leadng to a negatve correlaton. On the other hand a small quantzaton step sze ndcates that the qualty of codng the frame wll be good, thereby basng the decson towards codng the frame and hence leadng to a postve correlaton coeffcent. Among these features we can see that the sze of moton vectors and the HFE have the largest correlaton coeffcent values across most sequences and most rates. They are also relatvely uncorrelated wth a correlaton coeffcent.43. Hence, we choose these as representatve features for out test. The moton vectors are expensve to compute but we can replace them wth moton vectors from the prevous frame, as the correlaton between them s qute large at.87. The HFE for any frame of the sequence needs to be evaluated only once as ths can be stored and looked up every tme we code the sequence rrespectve of the target rates. We tran the densty functons, as descrbed n Secton II, for these selected features and buld our classfcaton based mode decsons. We evaluate these mode decsons over three dfferent sequences, Foreman, Coastguard and Slent. All these sequences are CIF at a frame rate 3 Hz. Of these sequences, Foreman s a hgh moton sequence, Coastguard s a medum moton sequence and Slent s a low moton sequence. The results are evaluated usng rate dstorton curves. Four dfferent target rates (5, 3, 45 and 6 kbps) were chosen for the test. The dstorton s measured usng the qualty metrc that we descrbed earler. We also compare the rate control usng these mode decsons wth one that we call No Skp rate control. Ths scheme tres to control the rate by only changng the quantzaton step sze and skps a frame only when there are no bts avalable to code t. These results are shown n the followng fgures.

20 Foreman Sequence No Skp Instantaneous Exhaustve Instantaneous Classfer.22 Dstorton Bt Rate (kbps) Fgure 9. Rate Control results for Foreman sequence.3.25 Coastguard Sequence No Skp Instantaneous Exhaustve Instantaneous Classfer Dstorton Bt Rate (kbps) Fgure. Rate control results for Coastguard sequence.2.8 Slent Sequence No Skp Instantaneous Exhaustve Instantaneous Classfer.6 Dstorton Bt Rate (kbps) Fgure. Rate Control results for Slent Sequence

21 We can see from these curves that rate control usng both the exhaustve as well as the classfcaton based mode decson perform better than the No Skp rate control as they provde smaller dstorton at the same target rate. The performance of the classfcaton based mode decson s close to the exhaustve search based decson. The error probablty for the classfer for Foreman s.7, for Coastguard s.27 and for Slent s.3. Ths leads to the classfer curve for Foreman not beng as close to the exhaustve mode decson curve as t s for the other sequences. We can also see that the average percentage mprovement n qualty across all the rates over the No Skp rate control s smallest for the Foreman sequence. Ths s because Foreman s a hgh moton sequence, hence skppng a frame s worse n qualty than codng a frame a majorty of the tme, thereby leadng to fewer frames beng skpped. In terms of computaton requrements the exhaustve mode decson uses roughly 3.5 tmes the computaton as the classfcaton based mode decson. The encoder wth the classfcaton based mode decson has a computaton complexty wthn 5% of an encoder wth no rate control strategy. IV. Mode Decson for Skppng or Codng Frames: Look-Ahead Case We extend the mode decson n the prevous secton by allowng a one-step look-ahead before makng a decson. As n the prevous secton we frst descrbe the exhaustve approach followed by a descrpton of the features that we select and the results for our experments. The steps n the exhaustve approach usng look-ahead are as follows. We frst skp the current frame and replcate the prevous decoded frame. The qualty and the rate for ths set of two frames { Xˆ ( n ), Xˆ ( n) = Xˆ ( n ) } are computed as descrbed before. Usng ths current reconstructed frame as reference, the future frame s both coded and skpped. Qualty and rate for the two frame sequences { X ˆ ( n) = Xˆ ( n ), Xˆ ( n + ) = Xˆ ( n ) } frame as well and { X ˆ ( n) = Xˆ ( n ), Xˆ ( n ) } skpped frame as reference, are computed. +,whenweskpthefuture, when we code the future frame usng ths We then code the current frame and qualty and rate for { ˆ ( n ), Xˆ ( n) } X are computed. Then usng ths coded frame as reference the next frame s both skpped and coded. Qualty and rate for { X ˆ ( n), Xˆ ( n + ) = Xˆ ( n) }, when we skp the future frame and { X ˆ ( n), X ˆ ( n + ) }, when we code the future frame, are also computed. Ths process may be represented pctorally as n Fgure 2.

22 Xˆ ( n ) skp Xˆ ( n ) skp code Xˆ ( n + ) Xˆ ( n ) code skp X ˆ ( n ) X ˆ ( n ) code Xˆ ( n + ) Tme Index n cost ( q Fgure 2. All frames generated for look-ahead exhaustve decson The decson on codng or skppng the current frame s made after lookng at the total + λr ) for each of the four paths (skp, code), (skp, skp), (code, skp) and (code, code). The path that provdes the best cost s dentfed and the decson for the current frame s made approprately. Ths strategy s very smlar to the Vterb decodng scheme wth a one step lookahead. The smlarty s shown n Fgure 3. c A c 3 Skp c 5 n c 2 c 4 B c 6 n n+ Code Fgure 3. Smlarty wth Vterb Decodng Scheme We look ahead one step whle tryng to make a decson at the current tme nstant n,.e. decde between nodes A and B. The costs compared.e. c + c3, c + c5, c 2 + c4 and c 2 + c6 + λr are c n the fgure correspondng to are compared to get the best cost. The node through whch ths path passes s chosen as the decson for the current frame whle the other node s dscarded. q

23 For the look-ahead classfer, the feature set that we start wth s the same as n the prevous secton. Of these features we fnd that the sze of moton vectors, the HFE and the quantzaton step sze are the most representatve features,.e. they have the largest correlaton coeffcents wth the decson sequence and have relatvely low correlaton between themselves. Of these features, dentfyng moton vectors requres a large amount of computaton, so we can replace the moton vector sze wth the moton vectors from the prevous frame. However, t s not good to approxmate the future frame moton vectors wth those from the prevous frame as the correlaton between the moton vectors that are two frames apart s not so large. Hence we decde aganst usng moton vector sze as a feature for ths decson strategy. As aganst ths, the quantzaton step sze for the current frame and those for future frames can be estmated usng the model we have, so we prefer to use ths. We choose four features for our classfer, a) Quantzaton step sze for current frame. b) Estmate of quantzaton step sze for future frame, f we skp the current frame. c) Estmate of quantzaton step sze for future frame, f we code the current frame. d) HFE for the current frame. All these features are easy to compute and as stated before, the HFE for any frame needs to be evaluated only once for any sequence. These look-ahead mode decsons are appled to the Foreman, Coastguard and Slent sequence and the results are compared wth the No Skp rate control. These results are shown n the followng fgures Foreman Sequence No Skp Look Ahead Exhaustve Look Ahead Classfer Dstorton Bt Rate (kbps) Fgure 4. Rate control results for Foreman sequence

24 .35.3 Coastguard Sequence No Skp Look Ahead Exhaustve Look Ahead Classfer Dstorton Bt Rate (kbps) Fgure 5. Rate control results for Coastguard sequence.2.8 Slent Sequence No Skp Look Ahead Exhaustve Look Ahead Classfer.6 Dstorton Bt Rate (kbps) Fgure 6. Rate control results for Slent sequence As before, we can see that rate control usng both the exhaustve as well as the classfcaton based mode decson perform better than the No Look rate control. The performance of the look-ahead mode decsons s better than the nstantaneous mode decsons. The error probablty for the classfer for Foreman s.68, for Coastguard s.22 and for Slent s.26 and hence the performance of the classfer s not as good as the exhaustve mode decson. As ponted out earler the average percentage mprovement n qualty across all the rates over the No Skp rate control s smallest for the Foreman sequence as t s a hgh moton sequence. In terms of computaton requrements the look-ahead exhaustve mode decson uses roughly 8x tmes the computaton as the classfcaton based mode decson. As before, the encoder wth the classfcaton based mode decson has a computaton complexty wthn 5% of an encoder wth no rate control. The number of steps that we are allowed to look-ahead can be

25 ncreased for a greater mprovement n performance, as the classfcaton based mode decson does not requre a sgnfcant ncrease the computaton requrements. Through our experments we have shown that we can use classfcaton based strateges to ntellgently decde between skppng a frame and codng t to acheve a better rate control strategy than just changng the quantzaton step sze to control rate and use frame skppng only when we have no bts avalable to code the current frame. All of ths dscusson was for rate control at the frame level, so we use one quantzaton step sze for the entre frame, however our classfcaton based schemes can easly be extended to macroblock layer rate control, when we can decde to skp or code a macroblock ntellgently. V. Extenson to Scalable Codng Scalable btstreams are used by vdeo codng schemes to mprove the error reslence over lossy networks. The btstream s parttoned nto multple layers and conssts of a base layer and one or more enhancement layers. The base layer s usually assgned the hghest prorty and error protecton and possesses enough nformaton for the decoder to reconstruct the vdeo sequence at a lower resoluton, frame rate or qualty. The enhancement layers consst of resdue nformaton between the base layer and the actual vdeo sequence thereby allowng for reconstructon of the vdeo at a hgher resoluton, frame rate or qualty. There are three dfferent scalabltes supported n the H.263 and MPEG-2 standards, whch are the Spatal, Temporal and SNR scalabltes. In the spatal scalablty, vdeo at a lower resoluton forms part of the base layer. In temporal scalablty, the base layer conssts of the vdeo sequence coded at a lower frame rate, whle n SNR scalablty the base layer conssts of the vdeo sequence coded at a hgh quantzaton step sze. In ths paper we consder two of these scalablty modes, temporal and SNR that are relevant to the skp or code mode decson mentoned earler. We focus our dscusson to the use of one enhancement layer. The work descrbed here can be extended to usng multple enhancement layers. Temporal scalablty s acheved by skppng frames whle codng the base layer, to obtan a lower frame rate vdeo. Frames that are skpped are predcted (may be forward, backward or b-drectonally predcted) from the current and prevous coded frames and the resdue and moton vectors are ncluded n the enhancement layer. SNR scalablty s acheved by codng frames at a hgher quantzaton step sze at the base layer and then codng the resdue between these frames and the actual vdeo at a lower quantzaton step sze to form the enhancement layer.

26 Usng the mode decsons descrbed n Sectons III and IV, we code a vdeo sequence by sometmes usng a hgh quantzaton step sze and sometmes skppng frames. Hence, ths coded vdeo may be vewed as a base layer generated usng a encoder that swtches between SNR and temporal scalabltes, as detaled later. We can therefore extend our work from the prevous secton to nvestgate the error reslence and performance of our strategy when mplemented over lossy networks. To do ths, we frst generate an enhancement layer correspondng to our base layer. Ths process s hghlghted n Fgure 7 and Fgure 8. Frames skpped n base layer Base Layer Predcton for skpped frames Enhancement Layer Orgnal Vdeo Fgure 7. Enhancement layer generaton for skpped frames From the fgure we can see that when we skp a frame n the base layer, we buld a predcton for the frame, that may be forward, backward or b-drectonally predcted from the precedng and followng coded frames and the resdue between ths predcton and the orgnal vdeo s ncluded as part of the enhancement layer. Ths s equvalent to temporal scalablty. The process of generatng the enhancement layer when we code a frame n the base layer s hghlghted n the followng fgure.

27 Base Layer Enhancement Layer Orgnal Vdeo Fgure 8. Enhancement layer generaton for coded frames Coded frames are subtracted from the orgnal vdeo and the resdue for each of these frames s ncluded as part of the enhancement layer, whch s coded at a lower quantzaton step sze. Ths s equvalent to SNR scalablty. Our codng scheme swtches between these two modes and we call t adaptve SNR/temporal (AST) scalable codng. Once we have bult the enhancement layer correspondng to our base layer, we code t at the same target rate as the base layer. In order to acheve the target rate, we change only the quantzaton step sze and do not allow for skppng of frames. We then smulate lossy network condtons and then reconstruct the vdeo sequence by combnng the two layers. The lossy condtons are smulated by throwng away some of the base layer macroblocks and some of the enhancement layer macroblocks and then combnng the layers. We examne dfferent error rates and ther mpact on the performance. Some error concealment s used at the decoder sde to mprove the qualty of the decoded vdeo. When a base layer macroblock s corrupted t s replaced by the correspondng macroblock from the prevous frame, whle when an enhancement layer macroblock s corrupted, t s thrown away. We also generate an enhancement layer for the No Skp rate control scheme (dentcal to SNR scalablty) and code and combne the layers as before. The resultng rate dstorton curves are plotted n Fgure 9.

28 .32.3 Foreman SNR AST Coastguard SNR AST.22.2 Slent SNR AST Overall Bt Rate (kbps) Overall Bt Rate (kbps) Overall Bt Rate (kbps) Fgure 9. Performance under lossy network condtons The curves plotted n the fgure are wth 5% loss n the base layer and % loss n the enhancement layer. As can be seen from the plots, the performance of the AST s better than usng just SNR scalablty across dfferent target rates for all the three sequences, wth only one excepton (Slent sequence at 2 kbps, 6 kbps for base layer and 6 kbps for enhancement layer). Sample frames from the Foreman sequence to hghlght the mprovement n dstorton are shown n Fgure 2. Fgure 2. Sample frames from Foreman sequence wth 5% base loss and % enhancement loss SNR Scalablty (left) and AST Scalablty (rght) The SNR scalablty frame has a PSNR of 27.7 db whle the AST scalablty frame has a PSNR of 29.9 db as compared to the orgnal frame. We also nvestgate the effect of varyng the enhancement layer and base layer losses at a fxed target rate of 3 kbps for base layer and 3 kbps for enhancement layer. We compare the performance wth SNR scalablty and the resultng mprovements are shown n Fgure 2.

29 % qualty mprovement (Foreman) % qualty mprovement (Coastguard) % qualty mprovement (Slent) %mprovement 4 2 %mprovement 2 %mprovement % enhancement layer loss % base layer loss % enhancement layer loss % base layer loss % enhancement layer loss %base layer loss 2 Fgure 2. Improvement n qualty of AST over SNR scalablty for dfferent error rates All of the above plots are generated for a target rate of 3 kbps for the base layer and the same for the enhancement layer. We can see that for all the dfferent error rates the performance n terms of qualty s better when we generate the base layer usng the classfcaton based mode decson as opposed to just changng the quantzaton step sze. We can also see that the percentage mprovement n qualty s hgher for Slent and Coastguard sequences as opposed to the Foreman sequence. Ths may be explaned by a combnaton of facts. When we skp a frame, the enhancement layer carres greater amount of nformaton for hgh moton sequences than t would for a low moton sequence. Also we have a larger amount of losses n the enhancement layer as compared to the base layer. So we lose more nformaton for hgher moton sequences when we skp frames n the base layer. Hence we have a smaller mprovement for the Foreman sequence. The case wth % loss n the base layer and % loss n the enhancement layer degenerates to the rate control problem that we focused on n Sectons III and IV. VI. Concluson The man contrbuton of ths paper s the classfcaton based approach to mode decsons n the vdeo encodng process. We successfully convert the problem of mnmzaton of a certan cost functon nto a standard mnmzaton of classfcaton error problem and use tradtonal pattern classfcaton technques to solve t. We use ths approach to mprove the performance of the Inter-Intra mode decson and reduce the btstream sze by 4.5~4.8% over the mode decson as provded n TMN. We then use ths approach to the rate control problem and show an mprovement n performance n the rate-dstorton sense over usng the no skp approach both for the nstantaneous as well as the look-ahead mode decson. The mprovement n qualty for the nstantaneous decson s 4~2% whle for the look-ahead decson t s 7~8% over the no skp rate control. We also extend ths work to the scalable vdeo codng and show that wth the