2 optmal per-pxel estmate () whch we had proposed for non-scalable vdeo codng [5] [6]. The extended s shown to accurately account for both temporal an

Transcription

1 Scalable Vdeo Codng wth Robust Mode Selecton Ru Zhang, Shankar L. Regunathan and Kenneth Rose Department of Electrcal and Computer Engneerng Unversty of Calforna Santa Barbara, CA 906 Abstract We propose to mprove the packet loss reslence of scalable vdeo codng. An algorthm for optmal codng mode selecton for the base and enhancement layers s developed, whch lmts error propagaton due to packet loss, whle retanng compresson effcency. We frst derve a method to estmate the overall decoder dstorton, whch ncludes the effects of quantzaton, packet loss and error concealment employed at the decoder. The estmate accounts for temporal and spatal error propagaton due to moton compensated predcton, and computes the expected dstorton precsely per pxel. The dstorton estmate s ncorporated wthn a rate-dstorton framework to optmally select the codng mode as well as quantzaton step sze for the macroblocks n each layer. Smulaton results show substantal performance gans for both base and enhancement layers. I. Introducton Scalable codng s an mportant tool for effcent transmsson of vdeo over packet swtched network. In a scalable coder, essental nformaton for the vdeo source s transmtted n the base layer, and can be decoded ndependently to obtan a coarse qualty of reconstructon. Supplementary nformaton s transmtted n hgher enhancement layers, whch, when combned wth base layer nformaton, mproves the vdeo reconstructon at the decoder. Syntax for scalable codng s provded n H.263+ and MPEG standards. Scalable vdeo codng offers means for robustness as base-layer reconstructon may be used as a fall-back opton n case of severe packet loss [1] [2]. For example, ATM networks can assgn hgher prorty n transportaton to the base-layer cells n case of congeston. In wreless networks, base-layer packets may be protected by stronger error correcton codes than enhancement-layer packets. However, n practce, some packet loss s nevtable even n the base-layer. Moreover, error propagaton wll amplfy the effect of packet losses n both base and enhancement layers, and wll further degrade the performance. In ths paper, we propose an optmal strategy for codng mode selecton per macroblock (MB) n both base and enhancement layers, whch substantally mproves the robustness of scalable vdeo codng systems. Whle there s a consderable volume of publshed work on mode selecton for packet loss reslence n the sngle-layer (non-scalable) vdeo codng (e.g., [3] [4] [5] [6]), very lttle work has been reported on the correspondng problem n scalable vdeo codng. We focus on an SNR scalable system, whch provdes layers wth the same spatal-temporal resoluton but dfferent reconstructon qualty. The key step n our dervaton s the estmaton of the overall decoder dstorton that takes nto account the quantzaton, packet loss, and the error concealment scheme. To calculate ths estmate, we extend the recursve

2 2 optmal per-pxel estmate () whch we had proposed for non-scalable vdeo codng [5] [6]. The extended s shown to accurately account for both temporal and spatal error propagaton, and to compute the total dstorton n each layer at pxel-level precson. For each MB, the predcton mode and quantzaton step sze are jontly selected to mnmze the rate-dstorton (RD) cost. Smulaton results show substantal gans n reconstructed vdeo PSNR at the base as well as enhancement layers. The paper s organzed as follows. In secton II, we derve the extended model that computes the optmal estmate of the overall dstorton of decoder reconstructon for each layer. We ncorporate the estmate wthn an RD framework for optmal selecton of mode and quantzer parameter n secton III. Secton IV presents smulaton results to demonstrate the performance of the method. II. Recursve Optmal per-pxel Estmate of Decoder Dstorton n Scalable Codng A. Prelmnares In the standard vdeo coder, the vdeo frame s segmented nto MBs. In the base layer, the MBs may be encoded n ether nter-mode or ntra-mode. In nter-mode, the MB s predcted" from the prevously decoded frame va moton compensaton, and the predcton error s encoded. In ntra-mode, the orgnal MB data s encoded drectly. In the enhancement layer, there are three possble predcton modes [7]. MBs can be predcted from the current base layer (upward), from the prevous enhancement layer (forward), or va combned predcton usng both (b-drectonal). The predcton resdue s then transform coded. Mode selecton s a powerful standard compatble tool to trade compresson effcency for packet loss reslence. The use of ntra-mode n the base layer, and upward predcton n the enhancement layer, can lmt error propagaton and s more effectve durng scene changes. However, n general they requre more bts for quantzaton. An optmal mode selecton strategy at the encoder should mnmze the overall dstorton n decoder reconstructon, whch ncludes the effects of quantzaton and packet loss, for the gven bt rate. Thus, a key task at the encoder s the estmaton of overall decoder dstorton. However, ths task s complcated by two factors. Spatal error propagaton beyond MB boundares (due to moton compensaton) can only be accurately accounted for by computng the dstorton per pxel. Further, dstortons due to quantzaton and packet loss are not addtve, but are nstead combned n a hghly complex fashon to produce the overall dstorton. In ths secton, we derve an algorthm to accurately estmate the total dstorton n decoder reconstructon at the dfferent layers of a scalable coder. We assume that the group of blocks (GOB) n each row s carred n a separate packet, and that the packets are ndependently decodable. Thus, the pxel loss rate equals the packet loss rate. We model the channel as a Bernoull process wth packet loss rate p b for the base layer, and packet loss rate p e for the enhancement layer. Note that ths model s assumed for presentaton smplcty, and more complex models may beconsdered as well. Let f denote the orgnal value of pxel n frame n, let ^f n n(b) and ^f n(e) denote ts encoder reconstructon at the base and enhancement layer respectvely. The reconstructed values at the decoder, possbly after error concealment, are denoted by f ~ n(b) and f ~ n(e). For the encoder, f ~ n(b) and f ~ n(e) are random varables. Assumng mean square error dstorton, the

3 3 overall expected dstorton for ths pxel, at the base and enhancement layers, s gven by d n(b) =Ef(f n ~ f n(b)) 2 g =(f n) 2 2f n Ef ~ f n(b)g + Ef( ~ f n(b)) 2 g: (1) d n(e) =Ef(f n ~ f n(e)) 2 g =(f n) 2 2f n Ef ~ f n(e)g + Ef( ~ f n(e)) 2 g: (2) We observe that the computaton of d n(b) and d n(e) requres the frst and second moments of the correspondng random varables, and develop recurson functons to sequentally compute these two moments. B. for the base layer It s easy to see that the problem of base layer mode selecton s dentcal to that of non-scalable codng. Thus, the algorthm derved n [5] [6] may be drectly appled for calculatng the total decoder dstorton. We brefly summarze the algorthm n ths subsecton. We assume, for presentaton smplcty, that the temporal error concealment technque s n use at the decoder. If the MB contanng pxel s lost, temporal replacement s used for error concealment,.e., the moton vector of ths MB s estmated as the medan of the moton vectors of the nearest three MBs n the prevous GOB (above). Let the estmated moton vector assocate pxel wth pxel k n the prevous frame. We thus have f ~ n(b) = f ~ n 1(b). k The probablty of ths event sp b (1 p b ). When the prevous GOB s also lost, the estmated moton vector s set to zero, and we have f ~ n(b) = f ~ n 1(b), wth probablty p 2 b. If the MB s correctly receved and has been ntra-coded, we have f ~ n(b) = ^f n(b) wth probablty (1 p b ). Thus, for a pxel n an ntra-coded MB, Ef ~ f n(b)g = (1 p b )( ^f n(b)) + p b (1 p b )Ef ~ f k n 1(b)g + p b 2 Ef ~ f n 1(b)g; (3) Ef( ~ f n(b)) 2 g = (1 p b )( ^f n(b)) 2 + p b (1 p b )Ef( f ~ n 1(b)) 2 k 2 g + p b Ef( f ~ n 1(b)) 2 g: If an nter-coded MB s correctly receved, the decoder has access to the quantzed resdue, ^e n(b), and the moton vector. Let the moton vector be such that pxel s predcted from pxel j n the prevous frame. The encoder's predcton s gven by ^g n(b) = ^f j n 1(b), and ts reconstructon s gven by ^f n(b) = ^e n(b) +^g n(b). The decoder must use ts predcton, ~g n(b) = ~ f j n 1(b). The correspondng reconstructon s gven by ~ f n(b) = ^e n(b) +~g n(b), wth probablty (1 p b ). As the decoder's predcton s not dentcal to encoder's predcton, error propagaton occurs even f the resdue s receved correctly. Thus, for a pxel n an nter-coded MB, Ef ~ f n(b)g = (1 p b )(^e n(b)+ef~g n(b)g) + p b (1 p b )Ef ~ f k n 1(b)g + p b 2 Ef ~ f n 1(b)g; Ef( ~ f n(b)) 2 g = (1 p b )Ef(^e n(b)+~g n(b)) 2 g (4) + p b (1 p b )Ef( f ~ n 1(b)) 2 k 2 g + p b Ef( f ~ n 1(b)) 2 g

4 4 = (1 p b )((^e n(b)) 2 +2^e n(b)ef~g n(b)g + Ef(~g n(b)) 2 g) + p b (1 p b )Ef( f ~ n 1(b)) 2 k 2 g + p b Ef( f ~ n 1(b)) 2 g: C. for the enhancement layer We now extend the algorthm to estmate the decoder dstorton at the enhancement layers. If an MB n the enhancement layer s lost, the decoder uses the correspondng baselayer block for error concealment. Let us denote the predcton value at the encoder sde as ^g n(e), and that of the decoder sde as ~g n(e). Let the transmtted resdue s denoted by ^e n(e). Note that ^g n(e) and ~g n(e) are not dentcal. Thus, even f the packet contanng the current pxel s receved correctly (wth probablty (1 p e )), the reconstructon at the encoder, ^f n(e) = ^e n(e) +^g n(e), s dfferent from the reconstructon at the decoder, f ~ n(e) =^e n(e)+~g n(e). Note that f ~ n(e) and ~g n(e) are random varables to the encoder. Thus, we have the followng recurson functons for the expected moments of f ~ n(e): Ef ~ f n(e)g = (1 p e )(^e n(e)+ef~g n(e)g) + p e Ef ~ f n(b)g Ef( ~ f n(e)) 2 g = (1 p e )Ef(^e n(e)+~g n(e)) 2 g (5) + p e Ef( ~ f n(b)) 2 g = (1 p e )((^e n(e)) 2 +2^e n(e)ef~g n(e)g + Ef(~g n(e)) 2 g) + p e Ef( ~ f n(b)) 2 g The expected moments of base layer are calculated as descrbed n the prevous secton. Let the moton vector of the MB assocate pxel wth pxel j n the prevous frame. The predcton, at the encoder and decoder, correspondng to the three predcton modes are gven by: for upward predcton: ^g n(e) = ^f n(b); ~g n(e) = ~ f n(b) (6) for forward predcton: ^g n(e) = ^f j n 1(e); ~g n(e) = ~ f j n 1(e): (7) for b-drectonal predcton: ^g n(e) = ( ^f j n 1(e)+ ^f n(b))=2; ~g n(e) = ( ~ f j n 1(e)+ ~ f n(b))=2: (8)

5 5 We reemphasze that these recursons are performed at the encoder n order to calculate the expected total dstorton at the decoder precsely per pxel. Whle for smplcty the recursons have been derved wthn a two-layer scalable codng setup, they can be extended n a straghtforward manner to compute the total decoder dstorton at each layer of a mult-layer scalable vdeo coder. Note that the estmate s precse for nteger-pxel moton estmaton. In the half-pxel case, the blnear nterpolaton makes the exact computaton of the second moment hghly complex. The estmate s approxmated by the smpler recurson of nteger-pxel moton compensaton. Further, for b-drectonal predcton, we assume Ef ~ f n(b) ~ f j n 1(e)g = Ef ~ f n(b)gef ~ f j n 1(e)g: (9) Although these approxmatons are sub-optmal, substantal gans are acheved. D. Smplfed for the specal case of guaranteed base layer An mportant practcal scenaro n scalable vdeo codng s when the base-layer packets are transmtted wth guaranteed recepton or wth neglgble packet loss rate. In ths case, the decoder reconstructon at the base-layer can be well approxmated by the encoder reconstructon,.e., now ~ f n(b) s not a random varable, but ~ f n(b) = ^f n(b). For ths specal case, we can use a smplfed to calculate the enhancement-layer dstorton. The recursons for the enhancement-layer can be rewrtten as: Ef ~ f n(e)g = (1 p e )(^e n(e)+ef~g n(e)g) + p e ^f n(b) Ef( f ~ n(e)) 2 g = (1 p e )Ef(^e n(e)+~g n(e)) 2 g (10) + p e ( ^f n(b)) 2 = (1 p e )((^e n(e)) 2 +2^e n(e)ef~g n(e)g + Ef(~g n(e)) 2 g) + p e ( ^f n(b)) 2 where the predcton, and, for the three predcton modes are gven by: for upward predcton: ^g n(e) =~g n(e) = ^f n(b): (11) for forward predcton: ^g n(e) = ^f j n 1(e); ~g n(e) = ~ f j n 1(e): (12) for b-drectonal predcton: ^g n(e) = ( ^f j n 1(e)+ ^f n(b))=2; ~g n(e) = ( ~ f j n 1(e)+ ^f n(b))=2: (13)

6 6 III. RD Optmzed Mode Selecton Algorthm for Scalable Codng We next ncorporate the dstorton estmate computed by the model nto an RD framework, and select the codng mode and quantzaton step sze of each MB to mnmze the overall decoder dstorton for the gven bt rate. The classcal" rate-dstorton problem s that of jontly selectng the codng modes for all the MBs to mnmze the total dstorton, D, subject to a gven rate constrant, R. Equvalently, we may recast the problem as an unconstraned Lagrangan mnmzaton, J = D + R, where s the Lagrange multpler. Note that ndvdual MB contrbutons to ths cost are addtve and, hence, the cost can be ndependently mnmzed for each MB. The codng modes are optmzed for the base and enhancement layers sequentally. For the base layer, the optmal mode and quantzaton step sze for each MB are chosen by the smple mnmzaton: mn (J MB(b)) = mn (D MB(b)+ b R MB (b)) (14) mode mode where the dstorton of the MB s the sum of the dstorton contrbutons of the ndvdual pxels: X D MB (b) = 2MB d n(b): (15) For the enhancement-layer, the predcton mode and quantzaton step sze are chosen to mnmze mn (J MB(e)) = mn (D MB(e)+ e R MB (e)) (16) mode mode where the dstorton of the MB s gven by: D MB (e) = X 2MB d n(e): (17) Note that we use the model to calculate the dstorton per pxel, whle the codng mode and quantzaton step sze are selected per MB va (14) and (16). The rate s controlled by usng the buffer status" to update b and e as n [6]. IV. Smulaton Results For the smulatons, we mplemented the -RD mode selecton strategy by approprately modfyng the UBC H.263+ codec wth two-layer scalablty [8]. The RTP payload format [9] s assumed for packetzaton, and each packet contans one GOB. A random packet loss generator s used to drop packets at a specfed loss rate. In the proposed system, the -RD algorthm s used for both layers for selecton of mode and quantzer parameter. For comparson, we use random ntra-update (RIU) [4] n the base layer, where MBs are randomly ntra-coded at the rate of 1=p b. In the enhancement layer, we compare the proposed scheme wth two standard approaches for predcton mode selecton. One method employs the quantzaton dstorton estmate () wthn an RD framework to make the selecton among the three predcton modes. In the other approach, only the upward predcton () mode s used. ensures that there s no error propagaton n the base-layer loss free case.

7 RIU (a) 28 (b) RIU (c) 27 Fg. 1. PSNR vs. enhancement layer bt rate (as a fracton of total rate). Base layer loss prone. Base layer methods: (proposed), RIU [4]; enhancement layer methods: (proposed),,. Base layer packet loss rate=5%, enhancement layer packet loss rate=15%. QCIF sequence carphone"(frame rate=10fps, total bt rate=100kbps): (a) base layer PSNR, (b) enhancement layer PSNR. CIF sequence LTS"(frame rate=15fps, total bt rate=600kbps): (c) base layer PSNR, (d) enhancement layer PSNR. (d) 250 frames from QCIF vdeo sequences carphone" and CIF vdeo sequence LTS" are compressed. The PSNR of lumnance reconstructon s computed for the sequence and averaged over dfferent channel smulatons (wth dfferent packet loss patterns). Fgure 1 shows the results for the QCIF sequence carphone" and CIF sequence LTS" when the packet loss rates n the base and enhancement layer are 5% and 15% respectvely. In the base layer, our proposed based mode selecton outperforms the RIU scheme by about 0.4ο1.0dB for carphone" and 0.6ο1.2dB for LTS". In the enhancement-layer, based robust mode selecton acheves PSNR gans of 0.9ο1.8 db for the carphone" sequence and 1.2ο2 db for the LTS" sequence over the competng methods. Ths corresponds to addtonal mprovement of 0.5ο0.8dB. Fgure 2 and Fgure 3 present the results when recepton of base layer packets s guaranteed. In ths case, base-layer performance s dentcal for both the methods of and RIU. Enhancement layer PSNR s shown versus packet loss rate n Fgure 2, and versus

8 % 10% 15% 20% enh. layer packet loss rate (a) 5% 10% 15% 20% enh. layer packet loss rate Fg. 2. PSNR vs. enhancement layer packet loss rate. Base layer loss free. Methods: (proposed),,. Enhancement layer bt rate rato=75%. (a) QCIF sequence carphone"(frame rate=10fps, total bt rate=100kbps), (b) CIF sequence LTS"(frame rate=15fps, total bt rate=600kbps). (b) (a) Fg. 3. PSNR vs. enhancement layer bt rate (as a fracton of total bt rate). Base layer loss free. Methods: (proposed),,. Enhancement layer packet loss rate=10%. (a) QCIF sequence carphone"(frame rate=10fps, total bt rate=100kbps), (b) CIF sequence LTS"(frame rate=15fps, total bt rate=600kbps). (b) enhancement layer bt rate (as a fracton of total rate) n Fgure 3. Note that the relatve performance of and depends on the packet loss rate and the enhancement layer bt rate. The proposed, however, consstently outperforms the other two methods. Note that smlar performance gans can be expected when proposed -RD mode swtchng algorthm s ncorporated nto other scalable vdeo codng schemes such as MPEG. V. Concluson We propose a method for optmal mode selecton n scalable vdeo codng, whch enhances robustness to packet loss. The method accurately estmates the overall decoder dstorton for each layer at pxel-level precson by accountng for quantzaton, error propagaton due

9 to packet loss, and error concealment scheme employed at the decoder. The estmate s then ncorporated wthn an RD framework for optmal mode selecton for macroblocks n each layer. Smulaton results show that the proposed method consstently outperforms conventonal mode selecton methods, and acheves sgnfcant PSNR gans n both base and enhancement layer. The algorthm requres no change to the codng syntax or to the decoder. Thus, t s compatble wth standards such as H.263+ and MPEG. Acknowledgement Ths work was supported n part by the Natonal Scence Foundaton under grant MIP , the Unversty of Calforna MICRO program, Csco Systems, Inc., Conexant Systems, Inc., Dalogc Corp., Fujtsu Laboratores of Amerca, Inc., General Electrc Co., Hughes Network Systems, Lernout & Hauspe Speech Products, Lockheed Martn, Lucent Technologes, Inc., Qualcomm, Inc., and Texas Instruments, Inc. References [1] R. Aravnd, M. R. Cvanlar and A. R. Rebman, Packet loss reslence of MPEG-2 scalable vdeo codng algorthms," IEEE Transactons on Crcuts and Systems for Vdeo Technology, vol.6, no. 5, Oct. 1996, pp [2] J. Vllasenor, Y.-Q. Zhang, and J.-T. Wen, Robust vdeo codng algorthms and systems," Proceedngs of the IEEE, vol.87, no.10, Oct pp [3] E. Stenbach, N. Farber and B. Grod, Standard compatble extenson of H.263 for robust vdeo transmsson n moble envronments," IEEE Trans. on Crcuts and Systems for Vdeo Technology, Vol.7, No.6, Dec. 1997, pp [4] G. Cote and F. Kossentn, Optmal ntra codng of blocks for robust vdeo communcaton over the Internet," Sgnal Processng: Image Communcaton, vol.15, No. 1-2, Sept. 1999, pp [5] R. Zhang, S. L. Regunathan and K. Rose, Optmal ntra/nter mode swtchng for robust vdeo communcaton over the Internet," Thrty-thrd Aslomar Conference on Sgnals, Systems, and Computers, Oct , [6] R. Zhang, S. L. Regunathan and K. Rose, Vdeo codng wth optmal ntra/nter mode swtchng for packet loss reslence," to appear on IEEE Journal of Selected Areas n Communcatons, specal ssue on Error-Reslent Image and Vdeo Transmsson. [7] ITU-T Recommendaton H.263, Vdeo codng for low bt rate communcaton," 1998 [8] H.263+ codec, [9] RTP Payload Format for the 1998 Verson of ITU-T Rec. H.263 Vdeo (H.263+)," Internet Draft, RFC2429,ftp://ftp.s.edu/n-notes/rfc2429.txt 9