A Full-Featured, Error Resilient, Scalable Wavelet Video Codec Based on the Set Partitioning in Hierarchical Trees (SPIHT) Algorithm

Transcription

1 A Full-Feature, Error Resilient, Salable Wavelet Vieo Coe Base on the Set Partitioning in Hierarhial Trees (SPIHT) Algorithm Sungae Cho an William A. Pearlman Center for Next Generation Vieo Researh Rensselaer Polytehni Institute 110 Eighth Street Troy, NY Corresponing Author: Prof. William A. Pearlman Tel: Fax: pearlman@rpi.eu Abstrat Compresse vieo bitstreams require protetion from hannel errors in a wireless hannel. The threeimensional (3-D) SPIHT oer has prove its effiieny an its real-time apability in ompression of vieo. A forwar-error-orreting (FEC) hannel (RCPC) oe ombine with a single ARQ (automati-repeat-request) prove to be an effetive means for proteting the bitstream. There were two problems with this sheme: the noiseless reverse hannel ARQ may not be feasible in pratie; an, in the absene of hannel oing an ARQ, the eoe sequene was hopelessly orrupte even for relatively lean hannels. In this paper, we eliminate the nee for ARQ by making the 3-D SPIHT bitstream more robust an resistant to hannel errors. We first break the wavelet transform into a number of spatio-temporal tree bloks whih an be enoe an eoe inepenently by the 3-D SPIHT algorithm. This proeure brings the ae benefit of parallelization of the ompression an eompression algorithms, an enables implementation of region-base oing. Then we emonstrate the paketization of the bitstream an the reorganization of these pakets to ahieve salability in bit rate an/or resolution in aition to robustness. Then we enoe eah paket with a hannel oe. Not only oes this protet the integrity of the pakets in most ases, but it also allows etetion of paket eoing failures, so that only the leanly reovere pakets are reonstrute. In extensive omparative tests, the reonstrute vieo is shown to be superior to that of MPEG-2, with the margin of superiority growing substantially as the hannel beomes noisier. Furthermore, the parallelization makes possible realtime implementation in harware an software. Keywors vieo ompression, vieo transmission, robust soure oing, error resilient transmission, ombine sourehannel oing, 3-D wavelet transform, embee wavelet oing. I. Introution Wavelet zerotree image oing tehniques were evelope by Shapiro (EZW) [2], an further evelope by Sai an Pearlman (SPIHT) [3], an have provie unpreeente high performane in image ompression with low omplexity. Improve two-imensional (2-D) zero-tree oing (EZW) by Sai an Pearlman [3] has been extene to three imensions (3-D EZW) by Chen an Pearlman [4], an has shown promise of an effetive an omputationally simple vieo oing system without any motion This material is base upon work supporte by the National Siene Founation uner Grant No. EEC , an IBM T. J. Watson Researh Center uner an IBM Faulty Awar. The government has ertain rights in this material.

2 2 ompensation, obtaining exellent numerial an visual results. Later, Kim an Pearlman evelope the three imensional SPIHT (3-D SPIHT) [5] oing algorithm improving on the 3-D EZW system of [4]. Wavelet zerotree oing algorithms are, like all algorithms prouing variable length oewors, extremely sensitive to bit errors. A single-bit transmission error may lea to loss of synhronization between enoer an eoer exeution paths, whih woul lea to a total ollapse of eoe vieo quality. Numerous sophistiate tehniques have been evelope over the last several eaes to make image transmission over a noisy hannel resilient to errors. One approah is to asae a SPIHT oer with error ontrol oing [7], [8]. The iea is to partition the output bitstream from the SPIHT oer into onseutive bloks of length N. Thentoeahblok heksum bits an m zero bits are ae to the en to flush the memory an terminate the eoing trellis at the zero state. The resulting blok of N + + m bits is then passe through a rate r rate-ompatible punture onvolutional (RCPC) oer [1]. However, this tehnique has the isavantage of still being vulnerable to paket erasures or hannel errors that our early in the transmission, either of whih an ause a total ollapse of the eoing proess. Another approah to proteting image bitstreams from bit errors is to restruture the noe test (NT) of the EZW algorithm. The approah is to remove epenent oing an lassify the oing bit sequene into subsequenes that an be protete ifferently using RCPC oes aoring to their importane an sensitivity. This type of tehnique was use by Man et al [9], [10]. Still another approah is to make image transmission resilient to hannel errors by partitioning the wavelet transform oeffiients into groups an inepenently proessing eah group. This metho was first reporte by Creusere [11] for use with the EZW algorithm. In reent work [6], Alatan et al. showe the embee image bitstreams an be elivere with error resiliene maintaine by iviing the bitstreams into three lasses. They protet the sublasses with ifferent hannel oing rates of the RCPC oer [1], an improve the overall performane against hannel bit errors. To ahieve robust vieo over noisy hannels, Kim et al. [14], [15] utilize the same RCPC oe as Sherwoo an Zeger [7], [8] with 3-D SPIHT, an foun that a single automati repeat request (ARQ) was also neessary to assure reliable reeption of the bitstream. ARQ, however, may not be feasible in ertain senarios an has the unfortunate onsequene of inreasing traffi on alreay ongeste hannels. In this paper, we first exten Creusere s work [11], [12], [13] to the 3-D SPIHT oer. We moify the 3-D SPIHT algorithm to work inepenently in a number of so-alle spatio-temporal (s-t) bloks, ompose of pakets that are interleave to eliver a fielity embee output bitstream. Therefore a bit error in the bitstream belonging to any one blok oes not affet any other blok. We then apply Kim et al. s metho [14], [15] of forwar error orretion, borrowe from Sherwoo an Zeger [7], [8], to every paket. Now we an etet eoing failures in any one paket an stop eoing, so that the rest of the blok s bitstream will not orrupt the orret bits alreay eoe up to that point. Beause this bitstream is embee an an s-t blok orrespons to a s-t region of vieo, the alreay eoe bits ontribute a less aurate renition of the region, while other regions orresponing to lean s-t bloks are reonstrute aoring to the rate of their bitstreams. This less sensitive soure oer substantially inreases hannel error robustness over a wie range of Bit Error Rates (BERs). In aition, we emonstrate that the oer has the funtionality of region-base ompression/eompression an spatial an/or temporal salability while retaining error resiliene. The organization of this paper is as follows: Setion 2 shows how to make the 3-D SPIHT bitstream more robust to hannel errors by breaking the wavelet transform into a number of spatio-temporal tree bloks. Setion 3 shows error resilient vieo transmission against hannel bit errors. Setion 4 ontains region-base ompression of vieo. Setion 5 provies omputer simulation results with omparisons to MPEG-2. Setion 6 onlues this paper.

3 3 II. Error Resilient 3-D SPIHT Vieo Compression Variable Length Coes (VLC) are use in urrent vieo oes for higher oing performane. Transmission bit errors for suh oes usually result in propagation of errors throughout the eoe file. In zerotree algorithms suh as SPIHT, when a single bit error ours in a bit onveying signifiane of a oeffiient or set of oeffiients, the result is loss of synhronization between the enoer an eoer, giving erroneously eoe ata beyon the point of the error. Therefore, a major onern of the esigner is the ontrol of errors so that reliable transmission an be obtaine. We esribe now a sheme, borrowe from Creusere s work with images [11], [12], [13], for partitioning a three-imensional wavelet transform into inepenent oing units, so that an error in any one unit oes not affet the others. We all this sheme Spatial an Temporal Tree Preserving 3-D SPIHT (STTP-SPIHT). A. System Overview a a b a b b Fig. 1. Struture of the spatio-temporal relation of 3-D SPIHT ompression algorithm Figure 1 shows how oeffiients in a three-imensional (3-D) transform are relate aoring to their spatial an temporal omains. Charater a represents a root blok of pixels (2 2 2), an haraters b,, enote its suessive offspring progressing through the ifferent spatial sales an numbers 1, 2, 3 label members of the same spatio-temporal tree linking suessive generations of esenants. We use 16 frames in a GOF (group of frames), therefore we have 16 ifferent frames of wavelet oeffiients. We an observe that these frames have not only spatial similarity insie eah one of them aross the ifferent sales, but also temporal similarity between frames, whih will be effiiently exploite by the Spatial an Temporal Tree Preserving SPIHT (STTP-SPIHT) algorithm. As shown in Figure 2 the basi iea of the error resilient 3-D SPIHT vieo ompression algorithm is to ivie the 3-D wavelet oeffiients into some number P of ifferent groups aoring to their spatial an temporal relationships, an then to enoe eah group inepenently using the 3-D SPIHT algorithm, so that P inepenent embee 3-D SPIHT substreams are reate. In this figure, we show an example of separating the 3- D wavelet transform oeffiients into four inepenent groups, enote by a, b,,, eah one of whih retains the spatio-temporal tree struture of normal 3-D SPIHT [5], an the normal 3-D SPIHT algorithm is just a ase of P =1. Eah substream has its own SPIHT heaer information. These heaers are most important for the reeiver to eoe the substreams orretly, an shoul be arefully protete from hannel errors. Furthermore, as we inrease the number of groups P, the heaer information overhea is also inrease,

4 4 a b a b aabbbbbb aabbbbbb a b a b bbbb bbbb aabbaabb aabbaabb bbbbbbbb bbb bb bbbb bbb bbbb bbbbbbbb aa aa aa aa aa aa aa aa b b b b bb bbbb bb bb bb bb bbbb bbbb bbbbbb bbbb bbbb bbbbbbbb bbbbb bb b bbbb bbbb heaer 1 heaer 2 substream 1 substream 2 heaer 3 substream 3 heaer 4 substream 4 STTP-SPIHT Global Heaer STTP-SPIHT Stream Fig. 2. Struture of the spatio-temporal tree preserving 3-D SPIHT(STTP-SPIHT) ompression algorithm beause the size of heaer information is fixe. In the ase of lower bit rates, the inreasingly signifiant size of overhea information ompare to ata information will be etrimental to vieo quality. To minimize these eleterious sie effets, we use a global heaer as shown in Figure 3. As we an see in this figure, the values in the shae areas (subim.x, subim.y, subim.z, pel bytes, smoothing) are the same for all the STTP-SPIHT heaers, beause eah substream orrespons to a region with the same size an bit rate. Therefore we an put the ommon variables to the beginning as a global heaer, an ontinue to write the information that is ifferent in other substreams, namely, threshol, mean shift, anmean. The final shape of the global heaer is shown in bottom of Figure 3. Using this iea, we an protet the SPIHT heaer information more effetively, an in aition to that, we an use more bits to enoe vieo ata in the same bit rate. STTP-SPIHT Heaers subim.x subim.y subim.x subim.y subim.z threshol pel_bytes smoothing mean_shift mean heaer for substream # 1 subim.x subim.y subim.x subim.y subim.z threshol pel_bytes smoothing mean_shift mean heaer for substream # 2 subim.x subim.y subim.x subim.y subim.z threshol pel_bytes smoothing mean_shift mean heaer for substream # n STTP-SPIHT Global Heaer subim.x subim.y subim.x subim.y subim.z pel_bytes smoothing threshol mean_shift mean threshol mean_shift mean Fig. 3. Struture of global heaer for the STTP-SPIHT ompression algorithm The P sub-bitstreams are then interleave in appropriate size units (e.g. bits, bytes, pakets,

5 5 Normal 3-D SPIHT STTP-SPIHT Stream # Stream # Stream # Stream #4 Final Bit Stream of STTP-SPIHT Fig. 4. Comparison of bitstreams between normal 3-D SPIHT an STTP-SPIHT with P =4 et.) prior to transmission so that the embee nature of the omposite bitstream is maintaine. Therefore we an stop eoing at any ompresse file-size or let run until nearly lossless reonstrution is obtaine, whih is esirable in many appliations inluing HDTV. Figure 4 illustrates the assembly of the final bitstreams of 3-D SPIHT an STTP-SPIHT with P = 4. The bitstreams are segmente into pakets numbere by orer of transmission. Enoing proees horizontally along the bitstreams, but in STTP-SPIHT, the transmission ours vertially ownwar an then from right to left along the bitstreams to aomplish interleaving. Therefore, the final STTP- SPIHT bitstream will be embee or progressive in fielity, but to a oarser egree than the normal SPIHT bitstream. After transmitting the pakets, the stream of normal 3-D SPIHT will be proesse sequentially at the estination. For the STTP-SPIHT, however, the interleave bitstream will be e-interleave, an eah substream will be proesse inepenently. By oing the wavelet oeffiients with multiple an inepenent bitstreams, any single bit error affets only one of the P streams, while the others are reeive unaffete. Therefore the wavelet oeffiients represente by a orrupte bitstream are reonstrute at reue auray, while those represente by the error-free streams are reonstrute at the full enoer auray. B. Color STTP-SPIHT We have onsiere only gray sale or one olor plane, luminane oing. In this setion, we will exten STTP-SPIHT to olor vieo oing while still preserving all the properties of the normal 3-D SPIHT. We follow the Kim et al. s [16] olor extension metho, an apply to eah s-t blok separately. We treat all olor planes of an s-t blok as one unit at the oing stage, an generate mixe YUV sub-bitstreams so that we an stop at any point of the bitstream an reonstrut the olor vieo of the best quality at the given bit rate. In our ase, the vieo sequene is YUV 4:2:2, where the U an V hrominane planes are half the size of the luminane Y plane as shown in Figure 5. The olor STTP-SPIHT algorithm is essentially the same as the gray sale STTP-SPIHT exept that now we have two more hrominane planes. Figure 5 shows the iea of error resilient olor STTP-SPIHT vieo ompression algorithm for YUV 4:2:2, whih is to ivie the 3-D wavelet oeffiients of the three planes into some number P ifferent groups aoring to their spatial an temporal relationships. We then enoe eah group inepenently with

6 6 Struture of the STTP-SPIHT Algorithm for Color YUV Format(4:2:2) aabbaabbbbbb a abb a a bb a aaabbbb a aaabbbb aa aa bb a aaabbbb aabb bb aabb bbbbbbbb bbbbbbbb bbbbbbbb bbbbbbbb a b a b a a bb a b a b a a bb a a bb a b a b a a a bb b a b a a b b a a bb a a b b a a bb a a b b a a bb a a b b a a bb a b a b a a bb a b a b a a bb a a bb a a bb a b a b a b a b a a bbaabb a a bbaabb a a bbaabb a a bbaabb Y U V a a aa aa aa aa aa bb bb bb bbbb bb bb bb bb bbbb bbbb bbbbbb bbbbbbbb b bbbbbbb b bbbbbbb b bbbb bbb Y Y Y Y a a a aa a a a a a a a U V U V U V U V b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b bbbb bbbb b b b b b bb b b b bb b b b b b b b b b b b b Stream 1 Stream 2 Stream 3 Stream 4 3-D Color SPIHT Bit Stream Fig. 5. Struture of the STTP-SPIHT Algorithm for Color YUV Format (4:2:2) the same rate using the 3-D SPIHT algorithm. C. Salability of STTP-SPIHT STTP-SPIHT is salable in rate, using blok interleaving/e-interleaving of the sub-bitstreams. In aition to that, it is highly esirable for STTP-SPIHT to have temporal an/or spatial salability for toay s multimeia appliations, suh as piture in piture funtion, vieo or volumetri image atabase browsing, an istane learning. The STTP-SPIHT oer is base on a multiresolution wavelet eomposition, so it shoul be simple to a the funtion of multiresolution eoing using the STTP-SPIHT sub-bitstreams. We an just partition the embee STTP-SPIHT sub-bitstreams into portions aoring to their subbans, an only eoe subbans orresponing to the esire resolution. Figure 6 shows partitionings of the STTP-SPIHT sub-bitstreams (P = 4) aoring to their orresponing spatial/temporal loations. In this figure, ark areas represent the low resolution of vieo sequene, an the other part is use for high resolution. As we an see, lower resolution information is usually loate at the beginning part of the sub-bitstreams. After oing some point of the vieo sequene, most of the remaining bit buget is use for oing the higher frequeny bans whih ontain the etail of vieo not usually visible at reue spatial/temporal resolution. Figure 7 illustrates this iea for a two layer ase of spatially sale Football sequene (frame 5). A low resolution vieo an be eoe from the first layer only, an the image size is If we use three layers,

7 7 STTP-SPIHT sub-bitstreams Sub-bitstream #1 Sub-bitstream #2 Sub-bitstream #3 Sub-bitstream # Fig. 6. Partitioning of the STTP-SPIHT sub-bitstreams into portions aoring to their orresponing spatial loations the lowest layer s size woul be Figure 8 is a typial example of multiresolution eoing of the Football an Susie sequenes. In this figure, (a) an (b) are spatial half resolution of the sequene, an () an () are full resolution of the sequene. The most valuable benefit of resolution salable eoing is saving of eoing time, beause the wavelet transformation onsumes most of the eoing time of the proess. For example, in a low resolution vieo of two spatial sales only, one-quarter of the number of wavelet oeffiients is transforme, while the spatio-temporal resolution of two spatial an temporal sales involves transformation of one-eighth of the number of wavelet oeffiients in the eoer. Therefore the lower resolutions of the sequenes nee muh less time to transform ue to the onsierably fewer number of wavelet oeffiients. III. Error Resilient Vieo Transmission In this setion, we ombine the blok interleaving sheme of STTP-SPIHT with the forwar error orreting oe of Kim et al. s work [14], [15]. Figure 9 illustrates the overall system with an optional ieal return hannel for ARQ. In our stuy, we shall make no use of ARQ. Kim et al. [14], [15] asae the 3-D SPIHT oer with RCPC using a single request ARQ strategy. Figure 10 shows the system esription of STTP-SPIHT/RCPC oer. In this figure, the RCPC oe stream is a segment of the hannel enoe bitstream. We first partition the STTP-SPIHT bitstream into equal length segments of N bits. In our ase, N = 200 bits. Eah segment of size N bits is then passe through a yli reunany oe (CRC) [17], [18] parity heker to generate = 16 parity bits. In a CRC, binary sequenes are assoiate with polynomials an oewors are selete suh that the assoiate oewor polynomials v(x) of N+ bits segments are multiples of a ertain polynomial g(x) alle the generator polynomial. Hene, the generator polynomial etermines the error etetion properties of a CRC. Next, m bits, where m is the memory size of the onvolutional oer, are pae at the en of eah N + bits segment to flush the memory of the RCPC oer. Hene, eah segment of N bits of the STTP-SPIHT bitstream is transforme into a segment of N + + m bits an passe through the rate r RCPC hannel enoer, whih is a type of punture onvolutional oer with the ae feature of rate ompatibility. The RCPC rate r is efine as k/n < 1, where k is the number of input bits entering the RCPC oer an n is the number of orresponing output bits. Hene, the rate an be interprete as the number of information bits entering the enoer per transmitte symbol. Finally, the RCPC oe stream is then transmitte over the omputer simulate binary symmetri

8 8 STTP SPIHT sub bitstreams Layere STTP SPIHT sub bitstreams Sub bitstream #1 Sub bitstream #2 Sub bitstream #3 Sub bitstream #4 Layere Sub bitstresm #1 Layere Sub bitstresm #2 Layere Sub bitstresm #3 Layere Sub bitstresm # Fig. 7. Multiresolutional eoer uses the higher resolution layer to inrease the spatial/temporal resolution of the vieo. hannel (BSC). Sine the enoer as reunant bits into 3-D SPIHT bitstream aoring to the rate r of the RCPC, the effetive soure oing rate R ef f is less than the total transmission rate R total,anisgiven by Nr R ef f = N + + m R total, (1) where a unit of R ef f an R total an be either bits/pixel, bits/se, or the length of bitstream in bits. As we saw before, Figure 4 graphially illustrates the final bitstream omparison between the normal 3-D SPIHT an the STTP-SPIHT. For STTP-SPIHT, we interleave by bloks of 200 bits aoring to Figure 4 to maintain embeeness. At the estination, the Viterbi eoing algorithm [19], [20] is use to onvert the pakets of the reeive bitstream into a STTP-SPIHT bitstream. In the Viterbi algorithm, the best path hosen is the one with the lowest path metri that also satisfies the heksum equations. In other wors, eah aniate trellis path is first heke by omputing a = 16 bit CRC. When the hek bits iniate an error in the blok, the eoer usually fixes it by fining the path with the next lowest metri. However, if the eoer fails to eoe the reeive paket within a ertain epth (here, that epth is 100) of the trellis, it stops eoing for that stream. The eoing proeure ontinues until either the final paket has arrive or a eoing failure has ourre in all P sub-bitstreams.

9 9 Fig. 8. Multiresolutional eoe sequene with STTP-SPIHT vieo oer (a) Top-left : spatial half resolution of Football sequene (frame 5) (b) Top-right : spatial half resolution of Susie sequene (frame 21) () Bottom Left : full resolution of Football sequene (frame 5) () Bottom right : full resolution of Susie sequene (frame 21). STTP- SPIHT Blok Interleaving CRC+RCPC Noisy Channel Viterbi Deoer + CRC Chek Deinterleaving STTP- SPIHT return hannel(optional) Fig. 9. STTP-SPIHT/RCPC system framework Figure 11 shows an example of eoing with P = 16 when eoing failure ours at paket number 67. In that ase, the normal 3-D SPIHT stops eoing at that point, but the STTP-SPIHT stops eoing only for the stream number 3, an ontinues to eoe the pakets of the other streams. After eoing, the normal 3-D SPIHT has only 66 lean pakets, but the STTP-SPIHT has more lean pakets, beause the normal 3-D SPIHT just stops eoing at the first eoing failure but the STTP- SPIHT an aept up to 16 eoing failures in the worst ase. In our example, substream number 3 has a eoing failure, an shorter length of bitstream after eoing an e-interleaving ompare to other substreams. The result is that the wavelet oeffiients resolution in substream number 3 are surroune by the other oeffiients of higher resolution in the other substreams. Therefore the reproution quality of the STTP-SPIHT is muh better than that of the normal 3-D SPIHT, beause STTP-SPIHT bit stream N m RCPC Coer RCPC Coe Stream N bit paket m bits of memory size bits for CRC Fig. 10. System esription of STTP-SPIHT/RCPC oer

10 10 STTP-SPIHT eoes many more lean bits ompare to the normal 3-D SPIHT. Normal 3-D SPIHT Deoing Fail Stop Deoing STTP-SPIHT Stream # Stream #2 Enoer Sie Stream # Stream # Deoing Fail Stop Deoing Stream # Stream # Stream #2 Deoer Sie Stream # Stream #16 Fig. 11. Example of eoing when eoing failure ours at paket number 67 IV. Region-Base Compression of Vieo with STTP-SPIHT We an take avantage of the STTP-SPIHT oer to get a region-base ompression of vieo sequenes. Using the oer, a speifi region of interest (ROI) gets reproue with higher quality than the rest of the image frame. Many lasses of vieo sequenes ontain areas whih are more important than others. It is unneessary an unwise to treat equally all the pixels in vieo sequene. To minimize the total number of bits, unimportant areas shoul be highly ompresse, thereby reuing transmission time an ost without losing the quality of the vieo sequene. One oul preserve the features with nearly no loss, while ahieving high ompression overall by allowing egraation in the unimportant regions terme regionally lossy oing or region-base lossy oing. Compression shemes, whih are apable for elivering higher reonstrution quality for the signifiant portions, are attrative in some ases suh as vieo transmission over hannels with highly onstraine banwith an volumetri meial image areas, where otors are intereste only in a speifi portion whih might ontain a isease. In the STTP-SPIHT, eah substream orrespons to a ertain region. Therefore, we an assign more bits to the substream, whih has the information of the region of interest, an assign the remaining bits of bit buget to the other substreams. Therefore, the sequene belonging to the bakgroun an the ROI s are oe inepenently at the speifie bitrates.

11 subban for ROI level subban substreams for ROI substreams for Bakgroun subban for Bakgroun Fig D Enoer System Configuration Fig Football sequene (frame15) (a) Top : Original sequene, (b) Bottom-left : 3-D SPIHT ompresse at bpp, () Bottom-right : region-base STTP-SPIHT result, requiring an overall rate of bpp

12 12 When we enoe vieo sequenes for region-base oing, we enter more information to the enoer, suh as the axis of top-left an bottom-right positions for the region of interest of the image, esire bit rate for the region of interest, an total bit rate or bakgroun bit rate, so that we an eie whih bitstreams woul orrespon to the region. There are two ways to eie the bakgroun bit rate. One is to speify the total bit rate an esire bit rate of the ROI, an the other is to speify the ROI bit rate an bakgroun s. In the first ase, we an always meet the target bit rate beause we an assign the remaining bit buget of total bit rate after assigning to the ROI, an the seon ase, we an hanle the quality of bakgroun image while maintaining the quality of the ROI. To the eoer sie, there are two kins of substreams, one for the ROI an the other for the bakgroun. The STTP-SPIHT eoes the sub-bitstreams inepenently. The only ifferene from the original STTP-SPIHT is that there are two kins of bit rates speifie by the enoer. After their inepenent eoings, the eoe wavelet oeffiients are reorere aoring to their spatial an temporal relationships, an then the inverse wavelet transform is applie. In Figure 12, the ROI is shown as the shae area. We an easily figure out whih sub-blok of oeffiients orrespons to the ROI, beause of the spatio-temporal relationships of eah blok. Then we assign more bits to the blok whih has the information of the ROI, an the remaining bits to the other bloks. As we an see in this figure, the final substreams for ROI are longer than the other substreams. Figure 13 shows an example of region-base oe vieo sequene. In this figure, (a) is the original Football sequene (frame 15) an (b) is the eoe image with an overall bit rate of bpp an () is region-base oe image with the same overall bit rate of bpp. In (b), the player s bak number an name are har to isern, but in (), they are muh learer. V. Results In this setion, we provie simulation results an ompare the propose STTP-SPIHT vieo oe with MPEG-2 in several aspets suh as soure oing in noisy an noiseless hannels, an ombine soure an hannel oing in noisy hannels. A. Robust Soure Coing In our test of error resiliene, we assume that the hannel is binary symmetri (BSC) with transition error probability ɛ. For MPEG-2, we use 15 frames in a GOP (Group Of Pitures), an the I/P frame istane is 3 (IBBPBBP...). For the STTP-SPIHT, sixteen frames in a GOF (Group Of Frames) are use, an a yai three level transform using 9/7 biorthogonal wavelet filters [21] is applie to the image, an possible robust partitionings are evaluate. We use one global heaer as shown in Figure 3 instea of using sub-heaers for eah substream, an we assume the SPIHT global heaer is not orrupte from bit errors. We interleave the streams in 200 bit pakets to maintain embeeness. Figure 4 ompares the enoe final bitstreams of the normal 3-D SPIHT an the STTP-SPIHT. The reeiver e-interleaves the bitstream to a series of substreams, eah one of whih is eoe inepenently. The algorithm is then teste using the monohrome Football (frame number 0-47) an Susie (frame number 16-63) sequenes. The istortion is measure by the peak signal to noise ratio (PSNR) ( ) PSNR =10log 10 B, (2) MSE where MSEenotes the mean square error between the original an reonstrute image sequenes. All PSNR s reporte for noisy hannels are averages over fifty (50) inepenent runs. The goal of the tests is to emonstrate the inherent resiliene of the STTP-SPIHT bitstream to hannel errors. Therefore, there has been no attempt to error onealment through postproessing either for STTP-SPIHT or MPEG-2 in these tests.

13 13 Fig Football sequene (frame14) oe to 1.0 bit/pixel without any bit error. (a) Top-left : Normal 3-D SPIHT, PSNR = B, (b) Top-right : STTP-SPIHT (P = 4), PSNR = B, () Bottom-left : STTP-SPIHT (P = 16), PSNR = B, () Bottom-right : MPEG-2, PSNR = B In our simulation of error resilient vieo transmission without error orretion oing, we eoe the STTP-SPIHT an MPEG-2 bitstreams to the en of the reeive bitstreams regarless of hannel bit errors, sine there was no mehanism to announe hannel errors. Figure 14 shows the football sequenes without any bit error using normal 3-D SPIHT (a), STTP-SPIHT (P = 4) (b), STTP- SPIHT (P = 16) (), MPEG-2 (), where the PSNR s are B, B, B an B respetively. The suessively lower PSNRs for the STTP-SPIHT are mainly ue to the presene of suessively more overhea bits neee to emarate sub-bitstreams. Figure 15 shows the effet in the presene of bit errors (BER = 10 4 ) without error orretion. The first three images (a), (b), () represent the sequenes use with the STTP-SPIHT algorithm with P = 16, P =55anP = 110, where the PSNR s are B, B an B respetively an the visual results notieably improve as the number of bloks P inrease, an image () shows the sequene with the MPEG-2 algorithm with orresponing PSNR is B an many bloks baly orrupte by bit errors. In Figure 16, BER = 10 5 is use with STTP-SPIHT with P =16(a),P =55(b)anP = 110 () an MPEG-2 algorithm (), an the orresponing PSNRs are B, B, B an B respetively. Here, at the lower error rate, even P = 16 bloks offers a eent reonstrution an P = 55 an 110 bloks ahieves a very goo reonstrution, in ontrast to the MPEG-2 eoe sequene where some bloks are ompletely obliterate. Figure 17 represents the frame by frame omparison of PSNR s of Football sequene with ifferent BERs an oe with 1.0 bit/pixel. The soli line on top shows the PSNR values of STTP-SPIHT (P = 16) without any bit errors, an the seon an the thir soli lines mean PSNR values with hannel bit errors, an the orresponing BERs are 10 5 an 10 4 respetively. The otte lines represent STTP-SPIHT with P = 55, an the ashe line iniates the MPEG-2 oe sequene. As we an see, the PSNRs for STTP-SPIHT without bit errors are similar to those of the MEPG-2, but in the

14 14 Fig Football sequene (frame15) oe to 1.0 bit/pixel with BER = (a) Top-left : STTP- SPIHT (P = 16), PSNR = B, (b) Top-right : STTP-SPIHT (P = 55), PSNR = B, () Bottom-left : STTP-SPIHT (P = 110), PSNR = B, () Bottom-right : MPEG-2, PSNR = B ase of hannel bit errors, the PSNR values of STTP-SPIHT (P = 55) are muh higher than those of the MPEG-2. Figure 18 illustrates the omparison of resulting average PSNR of Football sequene with wie range of BERs ( ) an ifferent number of bloks P (1, 4, 10, 16, 55, 110, 330), an oe with 1.0 bit/pixel. In this figure, when BER is very high (10 3 ), the average PSNR value of the STTP-SPIHT with P = 330 is still 2.41 B higher than that of the normal 3-D SPIHT in the ase of 100 times lower BER. In an error-free or very low bit error onition, the PSNR ifferenes are 0.77 B, 0.27 B, 0.08 B, 0.76 B, 1.00 B, 1.34 B with number of bloks P =4, 10, 16, 55, 110, 330, respetively. However, if the BER is larger than 10 6, the PSNR ifferenes are muh larger, ranging from 0.98 B to B epening on the number of bloks P,antheBERs. Table I shows the atual PSNR values for MPEG-2 an the ifferenes among MPEG-2, normal 3-D SPIHT an STTP-SPIHT (P = 4, 10, 16, 55, 110, 330) of Figure 18. Table II also shows the average PSNR values an the PSNR ifferenes of the Susie sequene. As we an see, the performane of the normal 3-D SPIHT is better than that of STTP-SPIHT in error free onition, then egraes rapily as the BER beomes higher. In the error free onition, the performane of STTP-SPIHT gets worse as P inreases ue to the heaer overhea in eah sub stream. Nevertheless, as the BER beomes higher, STTP-SPIHT with large P outperforms both normal 3-D SPIHT an MPEG-2 by signifiant margins. B. Combine Soure an Channel oing In our simulation of error resilient vieo transmission with error orretion apability, both the STTP-SPIHT an MPEG-2 bitstreams were protete ientially. As in previous works [7], [8], [14], [15], we protete the 200 bit pakets with the CRC, = 16 bit parity hek using generator polynomial

15 15 Fig Football sequene (frame14) oe to 1.0 bit/pixel with BER = (a) Top-left : STTP- SPIHT (P = 16), PSNR = B, (b) Top-right : STTP-SPIHT (P = 55), PSNR = B, () Bottom-left : STTP-SPIHT (P = 110), PSNR = B, () Bottom-right : MPEG-2, PSNR = B BER MPEG Normal 3-D SPIHT STTP-SPIHT (P = 4) STTP-SPIHT (P = 10) STTP-SPIHT (P = 16) STTP-SPIHT (P = 55) STTP-SPIHT (P = 110) STTP-SPIHT (P = 330) omits frames of faile eoing TABLE I Comparison of average PSNR(B) of Football sequene with ifferent BERs an number of bloks P (1, 4, 10, 16, 55, 110, 330) with STTP-SPIHT an MPEG-2 at total transmission rates of 2.53 Mbps. (no hannel oe, an oe to 1.0 bit/pixel)

16 STTP SPIHT (n = 16), BER = 0 STTP SPIHT (n = 55), BER = 0 MPEG 2, BER = 0 STTP SPIHT (n = 16), BER = 10 5 STTP SPIHT (n = 55), BER = 10 5 MPEG 2, BER = 10 5 STTP SPIHT (n = 16), BER = 10 4 STTP SPIHT (n = 55), BER = 10 4 MPEG 2, BER = Fig. 17. Comparison of frame by frame PSNR(B) of Football sequene with ifferent BERs (0, 10 5 an 10 4 )an oe to 1.0 bit/pixel with STTP-SPIHT an MPEG-2 without hannel oe MPEG 2 Normal SPIHT STTP SPIHT (P=4) STTP SPIHT (P=10) STTP SPIHT (P=16) STTP SPIHT (P=55) STTP SPIHT (P=110) STTP SPIHT (P=330) Fig. 18. Comparison of average PSNR(B) of Football sequene with ifferent BERs an number of bloks P (1, 4, 16, 55, 110, 330), an oe to 1.0 bit/pixel with STTP-SPIHT without hannel oe. g(x) =X 16 +X 14 +X 12 +X 11 +X 8 +X 5 +X 4 +X 2 +1, an RCPC hannel oer with onstraint length m = 6. We fouse on bit error rates (BER) of ɛ = 0. an 0.0, beause the BER s of most wireless ommuniation hannels are ɛ = The orresponing rates an R ef f are alulate from Equation(1). In our ase, we set the total transmission rate R total to 2.53 Mbps, r = 2/3 for ɛ =0. an 8/9 for ɛ = 0.0. For example, if we use a frames, the size of the bitstream is R total = 1, 351, 680 bits (equivalently total transmission rate of 1.0 bpp with frames), therefore Nr we have effetive number of pakets M 1 = R ef f /N = R (N++m)N total =(2/3)/ Nr pakets for ɛ = 0., an M 2 = R ef f /N = R (N++m)N total =(8/9)/ pakets for ɛ = 0.0. We teste Football an Susie sequenes of SIF ( ) format. For the STTP-SPIHT, we stop eoing for the substream where eoing failure ours. In our test, the path searh epth was

17 17 BER MPEG Normal 3-D SPIHT STTP-SPIHT (P = 4) STTP-SPIHT (P = 10) STTP-SPIHT (P = 16) STTP-SPIHT (P = 55) STTP-SPIHT (P = 110) STTP-SPIHT (P = 330) omits frames of faile eoing TABLE II Comparison of average PSNR(B) of Susie sequene with ifferent BERs an number of bloks P (1, 4, 10, 16, 55, 110, 330) with STTP-SPIHT an MPEG-2 at total transmission rates of 2.53 Mbps. (no hannel oe, an oe to 1.0 bit/pixel). set to 100, an if none of these paths is satisfie by the CRC, then the eoer stops eoing for the substream. For MPEG-2, whih is not embee an nees the full bitstream to see the whole frames, when eoing failure ours, we an use one of two shemes. One is just to use the orrupte paket, an the other is to put all 0 s to the orrupte paket. In this paper, we use the orrupte paket itself. For some trials with MPEG-2 in noisy onitions, eoing faile for several onseutive frames. For example in Figure 17, the bottom urve shows several frames at the beginning of the sequene with very low PSNR s. This phenomenon never ourre in any of the STTP-SPIHT runs. In the tables, we eie to omit faile eoing frames in the PSNR alulations for MPEG-2, an mark those PSNR s with asterisks. Table III shows the omparison of average PSNRs with STTP-SPIHT, normal 3-D SPIHT an MPEG-2 at total transmission rates of 1 Mbps an 2.53 Mbps of Football an Susie sequenes with bit error rates (BER) of 0, 0. an 0.0. In noiseless onitions, the PSNRs of STTP-SPIHT are similar to those of MPEG-2 or B higher. However, in the noisy hannel, we an see that the average PSNRs of the STTP-SPIHT with Football sequene are about 2-3 B higher at 2.53 Mbps an B higher at 1 Mbps than those of the MPEG-2, an in the ase of Susie sequene the average PSNRs of the STTP-SPIHT are about 2-3 B higher at 2.53 Mbps an 1-2 B higher at 1 Mbps than those of the MPEG-2. When we ompare with the normal 3-D SPIHT with ARQ, the average PSNRs of STTP-SPIHT are just 1-2 B lower. However, the ARQ strategy is often inappliable to real time senarios. Sine the STTP-SPIHT bitstream is embee, the eoer an request more information (aitional STTP-SPIHT/RCPC bitstream) to improve the vieo quality from the transmitter whenever more hannel banwith is available. Figure 19 shows the omparison of Football an Susie sequene with FEC an BER = 0.. Typial reonstrutions of Football an Susie sequene at total transmission rate of 2.53 Mbps an hannel bit error rates of 0. with P =16anMPEG-2areshowninFigure19(),()an (g), (h). As we an see, in Figure 19 (b) an (f), the normal 3-D SPIHT stops eoing when eoing failure ours, but in Figure 19 () an (g), the STTP-SPIHT stops eoing for the substream in whih eoing failure ours. Therefore any early eoing failure affets the full extent of the GOF in the normal 3-D SPIHT. However, the MPEG-2 eoe sequene in Figure 19 () an (h), the eoing failure affets some blok, an the blok is fille with some other piture s blok. Note in Figure 19 () an (g), the early eoing failure in STTP-SPIHT allows reonstrution of a small region at the bottom, right of enter (), an at the top, right of enter (g), with lower resolution only,

18 18 Sequene football Susie BER(Bit Error Rate) 0 0 STTP-SPIHT (P = 10) Mbps STTP-SPIHT (P = 16) MPEG D SPIHT STTP-SPIHT (P = 10) Mbps STTP-SPIHT (P = 16) MPEG D SPIHT BER(Bit Error Rate) STTP-SPIHT (P = 16)/RCPC Mbps MPEG-2/RCPC D SPIHT/RCPC D SPIHT/RCPC+ARQ STTP-SPIHT (P = 16)/RCPC Mbps MPEG-2/RCPC D SPIHT/RCPC D SPIHT/RCPC+ARQ omits frames of faile eoing TABLE III Comparison of average PSNRs (B) of Football an Susie sequenes with STTP-SPIHT, normal 3-D SPIHT an MPEG-2 at total transmission rates of 1 Mbps an 2.53 Mbps with bit error rates (BER) of 0, 0., 0.0 as only bits belonging to a low resolution were reeive orretly before essation of eoing. Full resolution regions, where all the bits were orretly eoe, surroun this reue-resolution region. Figure 20 ompares Football sequene between STTP-SPIHT with P =16anMPEG-2 sequene at total transmission rate of 2.53 Mbps an hannel bit error rates of 0.. In this figure, (a), (), (e), an (g) are typial results from STTP-SPIHT, an the frame numbers are 3, 7, 10, an 15 respetively, an (b), (), (f), an (h) are the MPEG-2 eoe sequene with the same frames as STTP-SPIHT s. As we an see, the eoing failure in the STTP-SPIHT affets a small region with lower resolution only. However, in the MPEG-2 eoe sequene, the eoing failure proues istortions in ranom positions, an the positions are marke with blak pixels or fille with another frame s image. Table IV shows the omparison of average PSNRs with STTP-SPIHT, normal 3-D SPIHT an MPEG-2 at total transmission rates of 1 Mbps an 2.53 Mbps of YUV 4:2:2 olor Football sequene with bit error rates (BER) of 0, 0. an 0.0. We ownloae the olor RGB football sequene from [22] an onverte it to YUV 4:2:2 format using the stanar filtering an ownsampling. This sequene ontains a ifferent sene from that of gray Football sequene. We an fin that the average PSNRs for Y of STTP-SPIHT are 0.5 to 1 B higher an U an V of STTP-SPIHT are 1 to 3 B higher than that of MPEG-2 in noisy hannel onitions. VI. Conlusions We have implemente a parallel SPIHT oing of vieo an have shown how robustness an resiliene to transmission errors an be ahieve in an embee vieo ompression algorithm with little inrease in its omplexity an little loss in noiseless hannel performane. The embee sub-bitstreams of the

19 19 Fig. 19. (a) Top-left : original Football sequene (frame15), (b) Top-right : Football reonstrution using normal 3-D SPIHT/FEC with BER = 0.. PSNR = 24.41B, () Seon row-left : Football reonstrution (frame15) using STTP SPIHT(P =16)/RCPC with BER = 0.. PSNR = 29.35B, () Seon row-right : MPEG-2/RCPC with BER = 0., PSNR = B, (e) Thir row-left : original Susie sequene (frame29), (f) Thir row-right : Susie reonstrution using normal 3-D SPIHT/RCPC with BER = 0.. PSNR = B, (g) Bottom-left : Susie reonstrution (frame29) using STTP- SPIHT(n=16)/RCPC with BER = 0.. PSNR = 38.53B, (h) Bottom-right : MPEG-2/RCPC with BER = 0., PSNR = B. Total transmission rate is set to 2.53 Mbps.

20 20 Fig. 20. (a) Top-left : Football reonstrution (frame 3) using STTP-SPIHT (P = 16)/RCPC with BER = 0.. PSNR = B, (b) Top-right : Football reonstrution (frame 3) using MPEG-2/RCPC with BER = 0.. PSNR = 21.24B, () Seon row-left : frame 7, PSNR = B, () Seon row-right : frame 7, PSNR = B, (e) Thir row-left : frame 10, PSNR = B, (f) Thir row-right : frame 10, PSNR = B, (g) Bottom-left : frame 15, PSNR = B, (h) Bottom-right : frame 15, PSNR = B. Total transmission rate is set to 2.53 Mbps.

21 21 Component Y U V BER(Bit Error Rate) STTP-SPIHT Mbps MPEG D SPIHT STTP-SPIHT Mbps MPEG D SPIHT BER(Bit Error Rate) STTP-SPIHT/RCPC Mbps MPEG-2/RCPC D SPIHT/RCPC D SPIHT/RCPC+ARQ STTP-SPIHT/RCPC Mbps MPEG-2/RCPC D SPIHT/RCPC D SPIHT/RCPC+ARQ omits frames of faile eoing TABLE IV Comparison of average PSNRs (B) of olor Football sequene (YUV 4:2:2) with STTP-SPIHT (P = 4), normal 3-D SPIHT an MPEG-2 at total transmission rates of 1 Mbps an 2.53 Mbps with bit error rates (BER) of 0, 0., 0.0 STTP-SPIHT algorithm allow the reorganization of the full bitstream to ahieve fielity an frame rate salability, but at a oarser level than in the full frame 3D-SPIHT algorithm. The STTP-SPIHT algorithm has prove to be muh more robust an resilient to hannel ranom bit errors than MPEG-2, whih it outperforms even in noiseless hannels. STTP-SPIHT uses no motion estimation/ompensation as oes MPEG-2, so is not suseptible to temporal propagation errors. Other features emonstrate are region-of-interest enoings an eoings an multiresolution eoing. In fat, the system is a fully operational olor vieo software oe with a parallel arhiteture suitable for harware realization. ACKNOWLEDGMENT The authors wish to express their gratitue to Young Seop Kim for his avie an enouragement, an his iea of how to inorporate region-of-interest oing into the overall system. Referenes [1] J. Hagenauer, Rate-ompatible punture onvolutional oes (RCPC oes) an their appliations, IEEE Transations on Communiations, vol. 36, pp , April [2] J. M. Shapiro, Embee image oing using zerotrees of wavelet oeffiient, IEEE Transations on Signal Proessing, vol. 41, pp , Deember [3] A. Sai an W. A. Pearlman A New,Fast an Effiient Image Coe Base on Set Partitioning in Hierarhial Trees, IEEE Trans. on Ciruits an Systems for Vieo Tehnology, vol. 6, pp , June [4] Y. Chen an W. A. Pearlman Three-Dimensional Subban Coing of Vieo Using the Zero-Tree Metho, SPIE Visual Communiations an Image Proessing, pp , Marh [5] B.-J Kim an W. A. Pearlman An embee wavelet vieo oer using three-imensional set partitioning in hierarhial trees, Pro. of Data Compression Conferene, pp , Marh [6] A. A. Alatan, M. Zhao, an A. N. Akansu Unequal Error Protetion of SPIHT Enoe Image Bit Streams, IEEE Journal on Selete Areas In Communiations, vol. 18, pp , June [7] P. G. Sherwoo an K. Zeger Progressive Image Coing for Noisy Channels, IEEE Signal Proessing Letters, vol. 4, pp , July 1997.

22 22 [8] P. G. Sherwoo an K. Zeger Progressive Image Coing on Noisy Channels, Pro. DCC, pp , April [9] H. Man, F. Kossentini, an M. J. T. Smith Robust EZW Image Coing for Noisy Channels, IEEE Signal Proessing Letters, vol. 4, no. 8, pp , August [10] H.Man,F.Kossentini,anM.J.T.Smith A Family of Effiient an Channel Error Resilient Wavelet/Subban Image Coers, IEEE Transations on Ciruits an Systems for Vieo Tehnology, vol. 9, no. 1, pp , February [11] C. D. Creusere A New Metho of Robust Image Compression Base on the Embee Zerotree Wavelet Algorithm, IEEE Transations on Image Proessing, vol. 6, no. 10, pp , Otober [12] C. D. Creusere Robust image oing using the embee zerotree wavelet algorithm, Pro. Data Compression Conferene, pp. 432, Marh [13] C. D. Creusere A family of image ompression algorithms whih are robust to transmission errors, Pro. SPIE, vol. 2668, pp , January [14] Z. Xiong, B.-J Kim, an W. A. Pearlman Progressive vieo oing for noisy hannels, In Pro. IEEE International Conferene on Image Proessing (ICIP 98), vol. 1, pp , Otober, [15] B.-J Kim, Z. Xiong, an W. A. Pearlman, an Y. S. Kim Progressive vieo oing for noisy hannels, Journal of Visual Communiation an Image Representation, vol. 10, pp , [16] B.-J Kim, Z. Xiong, an W. A. Pearlman Low bit-rate salable vieo oing with 3-D Set partitioning in Hierarhial Trees (3-D SPIHT), IEEE Trans. Ciruits an Systems for Vieo Tehnology, vol. 10, pp , Deember [17] T. V. Ramabaran an S. S. Gaitone A Tutorial on CRC Computations, IEEE Miro, vol. 8, pp , August [18] G. Castagnoli, J. Ganz, an P. Graber Optimum Cyli Reunany-Chek Coes with 16-Bit, IEEE Transations on Communiations, vol. 38, pp , January [19] G. D. Forney, Jr. The Viterbi algorithm, Pro. IEEE, vol. 61, pp , January [20] N. Seshari an C. Sunberg List Viterbi eoing algorithm with appliations, IEEE Transations on Communiations, vol. 42, pp , [21] M. Antonini, M. Barlau, P. Mathieu, an I. Daubehies Image oing using wavelet transform, IEEE Transations Image Proessing, vol. 1, pp , 1992 [22]