Incorporating Feature Point-based Motion Hypotheses in Distributed Video Coding

Transcription

1 Incorporatng Feature Pont-based Moton Hypotheses n Dstrbuted Vdeo Codng Ralph Hänsel, Henryk Rchter, Erka Müller Insttute of Communcatons Engneerng Unversty of Rostock Rostock, Germany {ralph.haensel, henryk.rchter, erka.mueller}@un-rostock.de Abstract Emergng moble vdeo applcatons ntroduce demands (e.g. low encodng complexty) whch do not ft well to conventonal vdeo codng (e.g. H.4/AVC, MPEG- 4). Upcomng codng schemes, followng the dstrbuted vdeo codng (DVC) paradgm, are well suted for moble vdeo encodng due to ther low encoder complexty. The performance of a DVC scheme s manly nfluenced by the Slepan-Wolf codng performance and the sde nformaton qualty. The sde nformaton s commonly obtaned by temporal nterpolaton, where fast nhomogeneous moton s stll challengng. In ths paper, fast moton s negotated by applyng a multple moton hypotheses pxel-based temporal nterpolaton method. Global and local moton estmaton are ncorporated. Smulaton results show sde nformaton qualty mprovements by up to 0.6 db for the sequence Stefan. Furthermore, state-of-the-art sde nformaton generaton approaches are outperformed by 1.2 db n terms of overall RD performance. I. INTRODUCTION Dstrbuted Vdeo Codng (DVC) s an nterestng topc, whch rases a strong research nterest snce the begnnng of the last decade. Despte extensve research actvtes a lot of challenges regardng to DVC are stll unsolved (e.g. feedback channel, source modelng, fast moton). Conventonal vdeo codng schemes (e.g. H.4/AVC) are very well suted for broadcastng scenaros due to ts hgh encodng and low decodng complexty. On the other hand, DVC s well suted for emergng applcaton scenaros requrng low encodng complexty (e.g. moble vdeo encodng). A low complexty vdeo encoder s acheved by shftng the complex moton estmaton part to the decoder. Furthermore, DVC s also capable to assure robust vdeo transmsson and effcent multvew vdeo codng. An extensve overvew of further applcaton scenaros for DVC s gven n [7], [10]. Dstrbuted Vdeo Codng (DVC) s based on the theores of Slepan and Wolf [12] as well as Wyner and Zv [15]. They prove that the rate dstorton performance of a codng scheme does not depend on the avalablty of sde nformaton (SI) at the encoder. The SI only needs to be avalable at the decoder. In terms of conventonal vdeo codng the SI s the moton compensated reference frame. In DVC, sde nformaton s commonly obtaned by temporal nterpolaton of adjacent key frames at the decoder. One of the unsolved challenges n DVC s hgh rate dstorton (RD) performance equvalent to conventonal nter frame vdeo codng (e.g. H.4/AVC). The RD performance on the one hand depends on the Slepan-Wolf (SW) codng performance. On the other hand, a hgh sde nformaton Y qualty s essental for hgh reconstructon qualty as well as low data rate. The sde nformaton s commonly estmated by temporal nterpolaton (sec. III). Fast moton s very challengng due to low temporal correlaton between adjacent frames. Ths problem cannot be solved by expandng the search range (max moton vector length mv ). Block based moton estmaton (ME) tends to msmatchng n case of a large search wndow. Thus, the necessary true moton s not obtaned. In ths paper, a multple moton hypotheses temporal nterpolaton method s proposed (sec. IV). It ncorporates multple ntal values for moton estmaton and a rather small search range. Therefore, the probablty of successful matchng and obtanng the true moton s ncreased. Three types of hypotheses are appled. The global moton hypothess reflects the camera moton, whereas the local moton hypothess corresponds to the moton of a segment. Fnally, a zero moton hypothess s also taken as ntal value. Smulaton results show an ncreased sde nformaton (SI) qualty as well as an mproved overall RD performance of 1.6 db and 1.2 db, respectvely (sec. V). Conclusons and some remarks on further work are gven n secton VI. II. PIXEL DOMAIN DVC SCHEME A pxel doman DVC (dstrbuted vdeo codng) scheme (fg. 1) s proposed n ths paper. It s chosen due to ts lower encodng complexty compared to DCT (dscrete cosne transform)-based DVC codng schemes. Furthermore, pxel doman DVC s essental for a flexble decodng process. In the frst step, the nput vdeo sequences s splt up nto key- and Wyner-Zv (WZ) frames. On the one hand, the key frames K are en- and decoded by an H.4 ntra en-/decoder. The key frame qualty s adjusted by the quantzaton parameter QP. On the other hand, each pxel of a WZ frame X s quantzed n the Wyner-Zv encoder. Therefore, a 2 M -step lnear quantzaton s appled. Subsequently, each bt plane q (b) of a quantzaton symbol q s Slepan-Wolf (SW) encoded. A bnary turbo encoder [11]

2 WZ Frames X Quant. q (1) q (M) Turbo Encoder Buffer Wyner-Zv Encoder Turbo Decoder ˆq (1) ˆq (M) Recon. Wyner-Zv Decoder ˆX splt M QP Y hmv zero hmv global Pxelbased ME/MC hmv local Global Moton Estmaton Local Moton Estmaton Sde Informaton Generaton Frame Buffer mux Key Frames K H.4 ntra Encoder H.4 ntra Decoder ˆK Fgure 1. Proposed pxel doman DVC scheme and Multple Moton Hypothess Pxel-based Temporal Interpolaton (M) s appled to mplement SW codng. The generated party symbols are stored n a buffer and send to the decoder on request. At the recever, the ntal sde nformaton Y s generated by temporal nterpolaton (sec. III). Fnally, the sde nformaton Y s corrected n the SW decodng and subsequent reconstructon process to obtan the decoded Wyner-Zv frame. III. SIDE INFORMATION ESTIMATION IN DVC A. Temporal nterpolaton Sde nformaton generaton n low complexty DVC, commonly explots the temporal correlaton. Fast moton s challengng for the block-based temporal nterpolaton (TI) appled n Bdrectonal moton estmaton wth spatal smoothng (BMESS [2]). Ths approach was extended by a herarchcal moton estmaton applyng dfferent block szes to cope wth fast moton (Moton compensated temporal nterpolaton MCTI [1], [3]). Fnally, block artfacts are a drawback of the mentoned algorthms. The later ssue s solved by Pxel-Based Temporal Interpolaton (, [13]). The fast moton challenge was negotated n [6] by estmatng a global moton model at the encoder, whch ncreases the encodng complexty. Fnally, mesh-based TI [8] performs well n deformaton areas. B. Temporal extrapolaton Fast moton s also very challengng for temporal extrapolaton (TX) approaches [4], [5]. Temporal extrapolaton do not requre small key frame dstances, due to ncorporatng decoded WZ frames. Moton estmaton s slghtly weaker compared to TI. Fnally, the TX approach shows good results n occluded areas. C. Spatal nterpolaton The spatal correlaton s exploted n [14] for sde nformaton generaton. At frst, a subset of pxel s decoded by ncorporatng temporal sde nformaton. Subsequently, a jont spatal and temporal sde nformaton method and SW decodng are appled to reconstruct the second subset of pxel. The spatal sde nformaton s estmated based on spatal nterpolaton. Thus, fast moton s less challengng, because spatal nterpolaton does not depend on hgh temporal correlaton (slow lnear moton). IV. MULTIPLE MOTION HYPOTHESES PIXEL-BASED TEMPORAL INTERPOLATION The am of the proposed sde nformaton generaton method (fg. 1) s to cope wth fast moton and thus the msmatchng problem. Therefore, the proposed Multple Moton Hypotheses Pxel-based Temporal Interpolaton (M) method ncorporates multple robust moton hypotheses and a rather small search range. These hypotheses nclude the global moton hmv global, the local moton hmv local and no moton hmv zero. They are ncorporated by the pxelbased moton estmaton/compensaton step, whch reduces the msmatchng probablty and block artfacts sgnfcantly. Fnally, the encoder should stay at very low complexty, by applyng the proposed algorthms only at the decoder sde. The proposed Multple Moton Hypotheses Pxel-based Temporal Interpolaton (M) approach ncorporates both adjacent key frames K 1, K +1 and follows these steps: 1) Global moton estmaton hmv global 2) Local moton estmaton hmv local 3) Forward pxel-based moton estmaton 4) Bdrectonal pxel-based moton estmaton 5) Moton compensaton

3 mv0,y mv 0 hmv local hmv global hmv zero SW 0 mv 0,x Fgure 4. Fnal search wndow SW 0 constructon k Fgure 2. l hmv local (a) adjacent feature ponts tr j,k,l Fgure 3. Moton of feature ponts j (b) hmv local for each pxel Local moton vector nterpolaton based on trangle mesh A. Global moton estmaton The global moton hypothess hmv global s obtaned by global moton estmaton. Therefore, feature pont matchng s appled to obtan robust moton vectors for characterstc areas (fg. 2). The SIFT [9] algorthm s used for feature pont fp detecton and matchng. Fnally, a vector medan flter (eq. 1) s appled on the set of feature pont moton vectors {mv }, obtanng a robust global moton estmaton by elmnatng outlers. hmv global mv j 2 j j B. Local moton estmaton mv mv j 2 (1) The local moton hypothess hmv local represents a more detaled hypothess compared to the global moton estmaton. Local moton s estmated based on the feature ponts fp and ts moton vectors mv estmated n the prevous step. Delaunay trangulaton s performed to dvde the frame nto trangles tr j,k,l (fg. 2), where addtonally feature ponts are placed at the frame border havng global moton. The local moton hypothess hmv local for each pxel s calculated by lnear nterpolaton of the moton vectors of the adjacent feature ponts (fp j, fp k, fp l, fg. 3(a)). Moton vectors are nterpolated by dsjont lnear nterpolaton of ts x- and y-components. Fnally, a dense ntal moton vector feld s obtaned (fg. 3(b)). C. Pxel-based moton estmaton Pxel-based moton estmaton (PBME) n conjuncton wth DVC was proposed n [13] to reduce block artfacts. In ths paper, the method s extended by havng a more adaptve non-rectangular search wndow SW 0 (fg. 4). The search wndow SW 0 s obtaned by the unon of the hypotheses search wndows (eq. 2, fg. 4). The zero moton search wndow SW zero for example s gven n equaton 3, where a small search range of SR = 6 s appled. The global moton and local moton search wndows SW global and SW local, respectvely, are n addton shfted by the correspondng moton hypothess hmv global or hmv local. SW 0 = SW zero SW local SW global (2) SW zero = {mv SR mv x SR SR mv y SR} (3) An equvalent rectangular search wndow covers a larger area. Thus, the complexty and probablty of msmatchng s reduced by applyng a non-rectangular search wndow. Subsequently, a dense moton vector feld s estmated by PBME. Therefore, a weghted block matchng approach s appled to obtan the moton vector mv 0 = (mv 0,x, mv 0,y ) (eq. 4, mv 0 SW 0 ). The neghborhood of each pxel (matchng wndow, MW ) n the prevous key frame K 1 s matched to the subsequent key frame K +1. The neghborhood s weghted by a Gaussan wndow (GW, σ GW = 5) and the moton estmaton s performed at full pxel accuracy. mv 0 = arg mn mv 0 (x,y) MW GW (x, y) K 1 (x, y) K +1 (x + mv 0,x, y + mv 0,y ) (4) D. Bdrectonal pxel-based moton estmaton Subsequently, the bdrectonal pxel-based moton estmaton (BPBME, eq. 5) s appled for moton feld refnement. Therefore, a search wndow SW 1 (eq. 6) algned wth the moton vector mv 0 s employed. Furthermore, a small search range SR = 1 s used to reduced the msmatchng probablty and computatonal complexty. The bdrectonal moton estmaton s performed at half pxel accuracy.

4 (a) BMESS [2] 19.8 db (b) [13] 20.1 db (c) proposed M 23.8 db (d) Orgnal Fgure 5. Soccer sde nformaton example, WZ-frame 61, QCIF, fps mv 1 = arg mn where mv 1 SW 1. mv 1 (x,y) MW GW (x, y) K 1 (x mv 1,x, y mv 1,y ) K +1 (x + mv 1,x, y + mv 1,y ) (5) SW 1 = {mv SR + mv 0,x /2 mv x SR + mv 0,x /2 SR + mv 0,y /2 mv y SR + mv 0,y /2} (6) E. Pxel-based moton compensaton Fnally, pxel-based moton compensaton (PBMC) at half pxel accuracy s appled, ncorporatng the neghborng key frames. The fnal sde nformaton Y s obtaned by averagng both moton compensated adjacent key frames K 1, K +1. V. SIMULATION RESULTS A comprehensve set of vdeo sequences (Coastguard, Foreman, Soccer, Stefan) s used to evaluate the performance of the proposed codng scheme. The well known BMESS, and the proposed M methods are compared n terms of sde nformaton qualty, RD-performance and computatonal complexty. Furthermore, QCIF resoluton sequences wth 15 fps and fps are used. The rate and dstorton values only rely on the lumnance component. A. Sde nformaton qualty The vsual sde nformaton qualty for the sequence Soccer at fps s shown n fgure 5. The result of the BMESS [2] sde nformaton generaton suffer from block artfacts and moton estmaton msmatchng (fg. 5(a)). The blockng artfacts are sgnfcantly reduced by applyng the [13] algorthm (fg. 5(b)). Fnally, the vsual and objectve qualty s sgnfcantly mproved applyng the proposed M approach (fg. 5(c)) due to reduced msmatchng probablty. Especally the nterpolaton qualty of fast movng objects s enhanced (background and rght border). However, the hghly nhomogeneous moton at the left frame border s stll challengng. Fast moton at the frame borders leads to an nherent problem of temporal nterpolaton. If an object BMESS proposed M soccer QCIF@fps WZ Frame BMESS proposed M stefan QCIF@fps WZ Frame Fgure 6. Sde nformaton qualty, Soccer, Stefan, QCIF, fps s only vsble n one frame, temporal nterpolaton wll not work, because the vsblty of the object n both adjacent frames s mandatory. Fgure 6 shows the sde nformaton PSNR (peak sgnal to nose rato) for each WZ frame of the sequences Soccer and Stefan at fps. The proposed M outperforms BMESS and especally for frames wth very fast moton (Soccer , Stefan ). However, the multple moton hypotheses approach mght be msleadng for a few low moton frames, where outperforms M by a very small margn (e.g. Stefan frame 80). Regardng to the average sde nformaton PSNR (tab. I), M outperforms state-of-the-art sde nformaton generaton methods (, MCTI) by up to 0.6 db. Very low moton sequences at hgh frame rates (e.g. Coastguard, Foreman) are slghtly challengng, because the hypotheses are sometmes msleadng.

5 foreman H.4ntra BMESS proposed M Rate [kbt/s] foreman QCIF@fps H.4ntra BMESS proposed M Rate [kbt/s] coastguard QCIF@15fps H.4ntra BMESS proposed M Rate [kbt/s] soccer QCIF@fps H.4ntra BMESS proposed M Rate [kbt/s] 25 stefan QCIF@15fps H.4ntra BMESS proposed M Rate [kbt/s] stefan QCIF@fps 25 H.4ntra BMESS proposed M Rate [kbt/s] Fgure 7. RD performance, quantzaton parameter set: (QP, M) = {(40, 1), (, 1), (, 1), (25, 2)} Table I AVERAGE SIDE INFORMATION QUALITY, PSNR [DB] QCIF BMESS MCTI M 15 fps [2] [13] [3] proposed Coastguard Foreman Soccer Stefan QCIF BMESS MI-2 M fps [2] [13] [4] proposed Coastguard Foreman Soccer Stefan values for MCTI and MI-2 are take for the correspondng lterature B. Overall RD performance The overall RD (rate-dstorton) performance of the DVC codng scheme (fg. 1) ncorporatng the proposed M s shown n fgure 7. The overall RD performance consders the H.4 ntra encoded key frames as well as the WZ frames reconstructon qualty and data rate. A fxed set of key frame quantzaton parameter (QP ) and WZ frame quantzaton parameter (M) was used. The proposed M method outperforms all mentoned sde nformaton generaton algorthm n terms of RD performance. For sequences wth low moton (e.g. Coastguard, Foreman) M shows a slghtly ncreased performance compared to. In most cases t shows better reconstructon qualty compared to the conventonal H.4 ntra codng scheme. The proposed M outperforms for fast moton sequences (e.g. Soccer, Stefan) by up 1.2 db. Furthermore, the proposed DVC codng scheme shows an RD performance close to H.4 ntra for Stefan at 15 fps, whch s remarkable for a DVC codng scheme. The sequence Soccer s stll challengng for DVC, because of ts very fast and nhomogeneous moton. C. Computatonal complexty The encodng complexty s very crtcal n a DVC codng applcaton. Therefore, the proposed M algorthm only affects the decodng complexty. The encodng complexty s retaned at ts low value. The DVC decodng complexty s domnated by the sde nformaton generaton and SW decodng complexty. and M are more complex than the block based BMESS method. The forward moton estmaton domnates the complexty of and M. Applyng a search range of SR = 10 n = 441 matchng wndow M W comparsons per pxel are requred. The search range s reduced to SR = 6 n M, due to ncorporatng multple moton hypotheses. Therefore, a maxmum number of = 507 matchng wndow comparsons per pxel are requred. Thus, the decodng complexty s only slghtly ncreased by M n the worst case (non overlappng search wndows) compared to. VI. CONCLUSIONS AND FURTHER WORK A Multple Moton Hypotheses Pxel-based Temporal Interpolaton (M) approach s proposed n ths paper. It negotate the challenge of fast nhomogeneous moton for sde nformaton generaton n dstrbuted vdeo codng (DVC). Therefore, local, global and zero moton hypotheses are ncorporated based on SIFT feature pont extracton and

6 matchng. The proposed M outperforms state-of-theart sde nformaton generaton n terms of sde nformaton qualty by up to 0.6 db and n terms of RD performance by up to 1.2 db. Very nhomogeneous moton and occlusons are stll challengng. Further work should nclude the complexty reducton of M. Furthermore, the temporal nterpolaton weakness at the frame border and n occluded areas should be solved by moton feld extrapolaton. REFERENCES [1] X. Artgas, J. Ascenso, M. Dala, S. Klomp, D. Kubasov, and M. Ouaret. The dscover codec: Archtecture, technques and evaluaton. In Proc. Pcture Codng Symposum (PCS), [2] J. Ascenso, C. Brtes, and F. Perera. Improvng frame nterpolaton wth spatal moton smoothng for pxel doman dstrbuted vdeo codng. In Proc. Speech and Image Processng, Multmeda Communcatons and Servces, Conference on (EC-SIP-M, EURASIP), [3] J. Ascenso and F. Perera. Herarchcal moton estmaton for sde nformaton creaton n wyner-zv vdeo codng. In Proc. of the 2nd nternatonal conference on Ubqutous nformaton management and communcaton (ICUIMC), [4] S. Borchert, R. Westerlaken, R. Gunnewek, and R. Lagendjk. On extrapolatng sde nformaton n dstrbuted vdeo codng. In Proc. Pcture Codng Symposum (PCS), [5] S. Borchert, R. Westerlaken, R. Gunnewek, and R. Lagendjk. Moton compensated predcton n transform doman dstrbuted vdeo codng. In Multmeda Sgnal Processng (MMSP), IEEE Workshop on, [6] F. Dufaux and T. Ebrahm. Encoder and decoder sde global and local moton estmaton for dstrbuted vdeo codng. In Proc. Multmeda and Sgnal Processng (MMSP), IEEE Workshop. on, [7] F. Dufaux, W. Gao, S. Tubaro, and A. Vetro. Dstrbuted Vdeo Codng: Trends and Perspectves. Image and Vdeo Processng, EURASIP Journal on, 2009:13, [8] D. Kubasov and C. Gullemot. Mesh-based motoncompensated nterpolaton for sde nformaton extracton n dstrbuted vdeo codng. In Proc. Image Processng (ICIP), IEEE Int. Conference on, [9] D. Lowe. Dstnctve mage features from scale-nvarant keyponts. Internatonal Journal of Computer Vson, 60:91 110, [10] F. Perera, L. Torres, C. Gullemot, T. Ebrahm, R. Leonard, and S. Klomp. Dstrbuted vdeo codng: Selectng the most promsng applcaton scenaros. Sgnal Processng: Image Communcaton, 23:9 2, [11] D. Rowtch and L. Mlsten. On the performance of hybrd fec/arq systems usng rate compatble punctured turbo (rcpt) codes. Communcatons, IEEE Transactons on, 48(6): , June [12] D. Slepan and J. Wolf. Noseless Codng of Correlated Informaton Sources. Informaton Theory, IEEE Transactons on, 19(4): , July [13] S. Sofke, R. Hänsel, and E. Müller. Human Vsual System aware Decodng Strateges for Dstrbuted Vdeo Codng. In Proc. Pcture Codng Symposum (PCS), Chcago, May [14] M. Taglasacch, A. Trapanese, S. Tubaro, J. Ascenso, C. Brtes, and F. Perera. Explotng spatal redundancy n pxel doman wyner-zv vdeo codng. In Image Processng (ICIP), IEEE Int. Conference on, [15] A. D. Wyner and J. Zv. The Rate-Dstorton Functon for Source Codng wth Sde Informaton at the Decoder. Informaton Theory, IEEE Transactons on, 22(1):1 10, Jan 1976.