A Fully SNR, Spatial and Temporal Scalable 3DSPIHT-Based Video Coding Algorithm for Video Streaming Over Heterogeneous Networks

Transcription

1 A Fully SNR, Spatial ad Temporal Scalable 3DSIHT-Based ideo Codig Algorithm for ideo Streamig Over Heterogeeous Networks Habibollah Dayali ad Alfred Mertis School of Electrical, Computer ad Telecommuicatios Egieerig Uiversity of Wollogog, Wollogog, NSW 2522, Australia School of Mathematics ad Natural Scieces, Istitute of hysics Uiversity of Oldeburg, Oldeburg, Germay Abstract ideo streamig over heterogeous etworks (e.g. the Iteret) requires high degree of scalability from the video codig. To achieve all types of scalability required for a fully scalable video codig, a modificatio of the 3DSIHT algorithm is preseted i this paper. The proposed algorithm, called Fully Scalable 3DSIHT (FS-3DSIHT), adds spatial ad temporal scalability features to the 3DSIHT algorithm, through the itroductio of resolutio depedet lists ad a resolutio-depedet sortig pass. The importat features of the origial 3DSIHT coder such as compressio efficiecy ad full embeddedess are kept. The output bitstream of the FS-3DSIHT ecoder cosists of a set of embedded parts related to the various quality, spatial ad temporal resolutios. It ca be easily reordered ad trucated by a simple parser i order to adapt various cliets requiremets (e.g. quality, spatial resolutio ad temporal resolutio) as well as badwidth variatio i a heterogeeous etwork. 1. INTRODUCTION For deliverig video iformatio over a heterogeous etwork such as the Iteret, a scalable video codig system is eeded to provide a bitstream that supports various of quality ad spatiotemporal resolutio. A scalable bitstream cosists of a set of embedded parts that offer icreasigly better sigal-to-oise ratio (SNR) or/ad greater spatial resolutio or/ad higher frame rate. These features are referred to as SNR scalability, spatial scalability ad temporal scalability respectively. Differet parts of a scalable bitstream ca be selected ad decoded by a scalable decoder to meet certai requiremets. Moreover, differet types of decoders with differet complexity ad access badwidth ca coexist. Traditioal video codig stadards such as MEG-2 ad H.263 provide some sort of scalability through a layer-based approach comprised of a base layer ad oe or more ehacemet layers. However, this approach provides oly coarse ad very limited meas of scalability ad results i a serious quality drop i compariso to the o-scalable sigle layer mode at the same rate [1]. A fie graularity scalability (FGS) optio adapted i MEG-4 oly icludes SNR ad temporal scalability ad oly for the ehacemet layer [2]. O the other had, 3D wavelet video codig schemes, e.g. [3 7], have a great potetial to support all types of scalability. This is due to the multiresolutio sigal represetatio offered by the 3D wavelet trasform. However, the trasform itself is ot the oly issue. It is particularly importat how the various spatiotemporal subbads i a 3D wavelet decompositio of a group of frame (GOF) are ecoded. The excellet rate-distortio performace ad scalable ature of the Set artitioig i Hierarchical Trees (SIHT) algorithm [8] for still images make it a attractive codig strategy also for video codig. A 3D extesio of SIHT for video codig has bee proposed by Kim et al. i [4, 6]. They applied a 3D wavelet trasform to a group of video frames (GOF) ad coded the wavelet coefficiets by the 3DSIHT algorithm. As reported i [4], eve without motio estimatio ad compesatio this method performs (i test with SIF moochrome 30 HZ sequeces) measurably ad visually better tha MEG-2, which employs complicated meas of motio estimatio ad compesatio. Although the 3DSIHT bitstream is tailored for full SNR scalability ad provides a progressive (by quality) bitstream, it does ot support spatial ad temporal scalabilities ad does ot provide a parsable bitstream that could be reordered accordig to desired spatiotemporal resolutios ad fidelities. I this paper, a fully scalable video codig scheme based o the 3DSIHT algorithm is preseted. The proposed algorithm is a extesio of our previous works [9,10] o scalable SIHT-based still image codig ad supports all spatial, temporal ad SNR scalability features together. The rest of this paper is orgaized as follow. Sectio 2 gives a overview of the video codec. Sectio 3 describes our modified algorithm, called FS-3DSIHT. The bitstream structure ad parsig process is explaied i Sectio 4. I Sectio 5, some experimetal results are preseted, ad fially, Sectio 6 cocludes the paper. 2. OERIEW OF THE IDEO CODEC Figure 1 shows the block diagram of the video codig system. O the ecoder side, the iput video sequece is first divided ito separate groups of frames (GOF). A 1D temporal filterig is first applied to each GOF, followed by a 2D spatial

2 !! Ecoder GOF Temporal Aalysis Spatial Aalysis FS-3DSIHT Ecodig Network Server Recostructed GOF at requested resolutios Temporal Sythesis Decoder / Cliet Spatial Sythesis FS-3DSIHT Decodig arser Figure 1. Block diagram of the video codig system. s-ll2 s-hl2 s-hl1 wavelet trasform. The umber of temporal decompositio ca be differet from the umber of spatial decompositio. Figure 2 shows a spatiotemporal multiresolutio structure of a decomposed GOF. The FS-3DSIHT ecoder the progressively ecodes the decomposed GOF from the lowest (coarsest) spatiotemporal bad to the highest bad i a way that the bits that belog to the idividual spatiotemporal bads are distiguishable i the output bitstream. Details will be give i the ext sectio. The decoder receives a reordered bitstream for each GOF which is tailored by a parser (trascoder) for the requested spatial resolutio, frame rate ad bit rate. The FS-3DSIHT decoder uses this bitstream to decode oly the required spatiotemporal bads. The iverse spatial ad temporal wavelet decompositio is the applied to the decoded spatiotemporal subbads to create a recostructed versio of the video sequece i the requested resolutio. 3DSIHT 3. THE FULLY SCALABLE 3DSIHT (FS-3DSIHT) ALGORITHM The 3DSIHT algorithm of [6] cosiders sets of coefficiets that are related through a paret-offsprig depedecy like the oe depicted i Figure 2. I its bitplae codig process, the algorithm deals with the wavelet coefficiets as either a root of a isigificat set, a idividual isigificat pixel, or a sigificat pixel. It sorts these coefficiets i three ordered lists: the list of isigificat sets (LIS), the list of isigificat pixels (LI), ad the list of sigificat pixels (LS). The mai cocept of the algorithm is maagig these lists i order to efficietly extract isigificat sets i a hierarchical structure ad idetify sigificat coefficiets, which is the core of its high compressio performace. Spatiotemporal Resolutios The 3D (1D temporal + 2D spatial) wavelet decompositio of a GOF, as illustrated i Figure 2, provides a multiresolutio structure that cosists of differet spatiotemporal subbads. I geeral, by applyig of 1D temporal decompositio ad of 2D spatial decompositio, of temporal resolutio ad of spatial resolutio are achievable. The total umber of spatiotemporal resolutios i this case is. To distiguish betwee s-lh2 s-lh1 s-hh2 s-hh1 Figure 2. A 3D wavelet decompositio of a GOF with 2 of both temporal ad spatial decompositio (21 spatiotemporal bads). The paret-offsprig relatioship betwee the wavelet coefficiets assumed by 3DSIHT is also show. differet "!$# resolutio, the spatiotemporal resolutio level is used to deote the spatial resolutio level ad temporal resolutio level #. The actual spatial ad temporal resolutios related to the level "!$# &(' ) ' are respectively % ad % of the spatial ad temporal resolutios of the origial sequece. Thus the lowest spatiotemporal resolutio (the lowest spatiotemporal frequecy subbad i the decomposed GOF) is referred to as level *+ +. The full spatiotemporal resolutio (the origial sequece) the becomes resolutio level. Spatiotemporal Subbad Sets A set of three spatial subbads,, s-hl&, s-lh&, s-hh& -, i temporal subbad level # that improve spatial resolutio of that temporal subbad from level + to level is defied as spatiotemporal subbad set level "!$# ad is referred to as. &(/ ). Note that for the maximum spatial resolutio level (i.e ),. &(/ ) oly cotais oe subbad which is the lowest spatial frequecy subbad (LL465 ) for all temporal subbad. To improve spatiotemporal resolutio from level!$# to level!$#, all spatiotemporal subbad sets that are located betwee these two resolutios eed to be added. These are. &(/ ) for 87 :9; ad # <7 #69#. A algorithm that provides full spatial ad temporal scalability would ecode the differet resolutio subbads separately, allowig a parser or a decoder to directly access the data eeded to recostruct a desired spatial or/ad temporal resolutio. The origial 3DSIHT algorithm of [6], however, sorts the wavelet coefficiets i such a way that the output bitstream cotais mixed iformatio of all subbads i o particular order, makig it impossible to trascode the bitstream without decodig it. t-l2 t-h2 y t-h1 t x

3 Codig Descriptio To add full spatial ad temporal scalability features to the 3DSIHT, the spatiotemporal subbad sets i the decomposed GOF eed to be coded separately. The fully scalable 3DSIHT (FS-3DSIHT) proposed here, fulfills this requiremet through the itroductio of multiple resolutiodepedet lists ad a resolutio-depedet sortig pass. It defies a set of LI, ad LIS lists for each. &(/ ), therefore there will be LI&(/ ), LS&(/ ), ad LIS&(/ ) for 0= >@?BAC!D >@?BA"E!FFFG! ad #0;#H>@?BAC!$#H>@?BAIE!FFFG! where >@?BA ad #H>@?BA are the maximum umber of spatial ad temporal resolutio respectively, supported by the ecoder. The FS-3DSIHT ecoder trasmits bitplae by bitplae ad defies the same paret-offsprig depedecy betwee the coefficiets i the 3-D wavelet pyramid as 3DSIHT (see Figure 2). I each bitplae, the coder starts ecodig from. &DJKLM/ ) JKL ad proceeds to. / /. I the resolutiodepedet sortig pass of the lists that belog to the. &(/ ), the algorithm first does the sortig pass for the coefficiets i the LI&(/ ) to fid ad output sigificace bits for all list etries ad the processes the LIS&(/ ). Durig processig the LIS&(/ ), sets that lie outside the spatiotemporal resolutio level "!$# are moved to their appropriate LIS related to the ext higher level of spatiotemporal subbad sets (i.e. LIS&(' / ) or LIS&(/ ) ' or LIS&(' / ) ' ). After the algorithm fiishes the sortig ad refiemet passes for resolutio level "!$# it will do the same for all other remaiig lists related to other spatiotemporal subbad set. The resolutio-depedet lists ca be scaed i such away that first all temporal resolutio be completed, the spatial resolutios, or vice versa. Accordig to the magitude of the coefficiets i the decomposed GOF, codig of the spatiotemporal subbad sets related to the the higher spatiotemporal resolutio usually starts from lower bitplaes. The FS-3DSIHT maages its multiple lists durig its resolutio-depedet sortig pass i a efficiet way such that the total umber of bits spet i a particular bitplae is the same as for 3DSIHT, but FS-3DSIHT arrage them accordig to their spatiotemporal resolutio depedecy i the bitstream. Note that the total storage requiremet for the LI&(/ ), LS&(/ ), ad LIS&(/ ) for all resolutios is the same as for the LIS, LI, ad LS used by the 3DSIHT algorithm. 4. BITSTREAM STRUCTURE AND SCALING Bitstream Structure Figure 3 shows the bitstream structure geerated by the FS- 3DSIHT ecoder. The bitstream is first divided ito differet parts accordig to the differet bitplaes. The bitplae codepart N8O deotes all bits obtaied at bitplae codig process level. Iside each bitplae codepart, the bits that belog to differet spatial subbad sets are separable, ad similarly, iside each spatial subbas set codepart, the bits that belog to differet temporal subbad sets come i order. The smallest codepart uit i the bitstream is referred to as NO &(/ ) i Fig- Bitplae Spatial subbad Temporal subbad... Header max max k max kmax-1 k k,l max k,lmax -1 k,l k,1 Figure 3. Structure of the FS-3DSIHT ecoder bitstream which is made up of differet quality, spatial ad temporal resolutio parts. NO &(/ ) deotes the codepart that obtai for codig the spatiotemporal subbad set level "!$# (i.e.. &(/ ) ) at bitplae level. ure 3. It is the codepart that belogs to spatiotemporal subbad set level "!$# (i.e.. &(/ ) ) at bitplae level. To support bitstream parsig, some markers are required to be put ito the bitstream to idetify the parts of the bitstream that belog to the differet spatial ad temporal resolutio ad bitplaes. Bitstream Scalig The flexible hierarchical structure of the FS-3DSIHT bitstream allows bitstream parsig to obtai sub-bitstreams for differet reduced spatiotemporal resolutios ad qualities, all from a sigle high bit rate, full resolutio bitstream. The parsig (reorderig) process is a simple scale reducig task i which oly the related codeparts of the mai bitstream that belog to the requested resolutio are selected ad ordered without eed to decode. For example, to provide a bitstream for resolutio level Q!$#RQM, i each biplae codepart, oly the codeparts of the spatial subbad sets that are located iside the spatial resolutio level Q are kept, ad similarly, i each selected spatial part, oly the temporal parts that fall iside the requested temporal resolutio level ( #SQ ) are kept, ad all other parts are removed. Therefore, UQ the selected codeparts i bitpale level are,mno &(/ )$T 7 7 >@?BAC!$#RQ 7 # 7 #H>@?BA -. Note that all marker iformatio for idetifyig the idividual bitplaes ad resolutio are oly used by the parser ad do ot eed to be set to the decoder. A distict feature is that, after parsig, the reordered bitstreams for all spatiotemporal resolutios are completely fie graular ad ca be trucated at ay poit to obtai the best recostructed sequece at the desired spatiotemporal resolutio ad bit rate. The decoder required for decodig the reordered bitstream exactly follows the ecoder, similar to the origial 3DSIHT algorithm. It eeds to keep track of the various lists oly for spatial ad temporal resolutio greater or equal to the required oe. Thus, the proposed algorithm aturally provides computatioal scalability as well. 0

4 Figure 4. Forema ad Hall Moitor sequeces at differet spatial resolutios. Table 1. Mea Y-SNR results for 240 frames of 30HZ CIF Forema ad Hall Moitor sequeces at differet spatial ad temporal resolutios ad bit rates obtaied by a FS-3DSIHT decoder. Spatial Temporal Rate Mea Y-SNR (db) resolutio resolutio (kbps) Forma Hall Moitor CIF CIF CIF QCIF QCIF QCIF QCIF QCIF QCIF QCIF QCIF EXERIMENTAL RESULTS To show the FS-3DSIHT ecoder s full scalability features, tests were performed o two CIF sequeces: Forema ad Hall Moitor. The origial frame rate of these sequeces is 30 frames per secod. The GOF size used by the ecoder is 16 ad for the 3D wavelet decompositio of each GOF, 3 of 1D decompositio by Haar filter i temporal directio followed by 4 of 2D spatial decompositio by 9/7-tap filters [11] were applied. The FS-3DSIHT ecoder was set to ecode 240 frames (15 GOFs) of each sequece at 600kbps ad to provide 5 of spatial resolutio ad 4 of temporal resolutio (total 20 spatiotemporal resolutios), which are the maximum possible resolutio accessible by the above metioed 3D decompositio. After ecodig, the FS-3DSIHT bitstream was fed ito a parser to produce progressive (by quality) bitstreams for differet spatial ad temporal resolutios. The bitstreams were decoded at various spatial resolutios (see Figure 4), frame rates ad bit rates. Table 1 shows the mea SNR results obtaied for the lumiace compoets of the decoded sequeces. Referece frames for lower resolutios were defied by takig the lowest frequecy subbad frames after applyig appropriate of temporal ad spatial wavelet decompositio to the origial sequeces. For full spatial ad temporal resolutio these results are comparable with the 3DSIHT coder of [4, 6], but our coder additioally yields the full scalability features. The SNR values i Table 1 show that high- SNR bitstreams ca be easily geerated at low rates with reduced spatio temporal resolutio. 6. CONCLUSIONS I this paper we preseted a fully scalable 3DSIHT algorithm (FS-3DSIHT) for video codig that supports all types of spatial, temporal ad SNR scalability together. The iterestig features of the origial 3DSIHT algorithm such as high compressio efficiecy ad rate embeddedess of the bitstream are kept. The FS-3DSIHT ecoder bitstream ca be easily reordered (parsed) to obtai rate embedded subbitstreams for various spatiotemporal resolutios without the eed of decodig. The proposed fully scalable video codec is a good cadidate for multimedia applicatios such as video iformatio storage ad retrieval systems, ad video streamig over heterogeous etworks where a wide variety of users eeds to be differetly serviced accordig to their etwork access ad data processig capabilities. 7. ACKNOWLEDGMENT The first author would like to ackowledge the fiacial support provided for him by the Miistry of Sciece, Research ad Techology (MSRT), Ira ad Kurdista Uiversity, Saadaj, Ira durig doig this research as a part of his hd study at the Uiversity of Wollogog, Australia. REFERENCES [1] L. Yag, F.C.M. Martis, ad T.R. Gardos, Improvig H.263+ scalability performace for very low bit rate applicatios, i roc. SIE-CI 99, Sa Jose, CA, Ja. 1999, vol. 3653, pp [2] W. Li, F. Lig, ad X. Che, Fie graularity scalability i MEG-4 for streamig video, i roc. IEEE It. Symp. Circuits ad Systems, 2000, vol. 1, pp [3] C. odichuk, N. Jayat, ad N. Farvardi, Threedimesioal subbad codig of video, IEEE Tras. Image rocessig, vol. 4, o. 2, pp , Feb [4] B.-J.Kim ad W. A. earlma, A embedded video coder usig three-dimesioal set partitioig i hierarchical trees (SIHT), i proc. IEEE Data Compressio Cof., Mar. 1997, pp [5] S.-J. Choi ad J. W. Woods, Motio compesated 3-d subbad codig of video, IEEE Tras. Image rocessig, vol. 8, o. 2, pp , Feb

5 [6] B.-J. Kim, Z. Xiog, ad W. A. earlma, Low bit-rate scalable video codig with 3-d set partitioig i hierarchical trees (3-D SIHT), IEEE Tras. Circ. ad Syst. for ideo Techology, vol. 10, o. 8, pp , Dec [7] S.-T. Hsiag ad J. W. Woods, Embedded video codig usig ivertible motio compesated 3-D subbad/wavelet filter bak, Sigal rocessig: Image Commuicatio, vol. 16, o. 8, pp , May [8] A. Said ad W. A. earlma, A ew, fast ad efficiet image codec based o set partitioig i hierarchical trees, IEEE Tras. Circ. ad Syst. for ideo Techology, vol. 6, pp , Jue [9] H. Dayali ad A. Mertis, Highly scalable image compressio based o SIHT for etwork applicatios, i roc. IEEE It. Cof. Image rocessig, Rochester, NY, USA, Sept. 2002, vol. 1, pp [10] H. Dayali ad A. Mertis, Fully scalable waveletbased image codig for trasmissio over heterogeeous etworks, i roc. 1st Workshop o the Iteret, Telecommuicatios ad Sigal rocessig, WITS 02, Wollogog, NSW, Australia, Dec. 2002, pp [11] M. Atoii, M. Barlaud,. Mathieu, ad I. Daubechies, Image codig usig wavelet trasform, IEEE Tras. Image rocessig, vol. 1, pp , Apr