CURVE-FITTING ALGORITHMS FOR SHAPE ERROR CONCEALMENT

Transcription

1 CURVE-FITTIG ALGORITHMS FOR SHAPE ERROR COCEALMET Xaohuan L ( + ), Gudo M. Schuster (*) and A. K. Katsaggelos ( + ) ( + )Image and Vdeo Processng Laborator, Department of Electrcal and Computer Engneerng, orthwestern Unverst, Evanston, IL 68 (*) Abtelung Elektrotechnk, Hochschule für Technk Rapperswl, Swtzerland ABSTRACT Introducton of Vdeo Objects (VOs) s one of the major contrbutons of MPEG-4 to the prevous MPEG standards. The a-plane of a VO defnes ts shape and hence determnes the boundar of ts teture. In error prone communcaton networks, shape nformaton, as well as teture nformaton s subject to bt error contamnaton. In ths paper, we propose two post-processng shape concealment technques for MPEG-4 vdeo sequences. Both technques operate b frst detectng contet for the mssng area; second etractng the geometrc nformaton of the contet; and thrd reconstruct the mssng contour usng a smooth curve. The frst technque uses an mplctl defned second order curve whle the second technque uses cubc Hermte splnes. Epermental results that show the superor performance of these methods are gven at the end of the paper.. BACKGROUD One of MPEG-4 s most promnent features compared to MPEG - and s the presentaton of vdeo objects (VOs) [][]. Based on ths, MPEG-4 vdeo frames are nterpreted and manpulated n terms of vsual objects. These objects are defned b the boundar of ther shape, whch s descrbed as a gra-scale or bnar alpha-plane (see an eample n Fgure ). In error-prone communcaton envronments, MPEG-4 vdeo packets are often coded n the data-parttonng mode, under whch shape/moton nformaton s separated from the teture moton. As specfed b the standard, f onl teture s lost, shape and moton can be tapped nto to conceal teture, whle f shape/moton s lost, the whole packet s dscarded. That s a basc motvaton to desgn a concealment technque especall for shape, besdes all the estent algorthms for teture concealment. The MPEG-4 standard has three [3] bult-n errorreslent technques for shape nformaton. ) To account for potental packet loss, MPEG-4 redefnes the contet of the CAE when the encoder enables the error reslence mode. Ths s accomplshed b denotng an pel locaton that s eternal to the current vdeo packet as transparent. Applcable to both nter- and ntra-cae modes, ths approach lmts the propagaton of shape errors n a nos channel. ) Data parttonng. Ths approach separates the MB header, bnar shape nformaton, and MVs from the teture nformaton. A resnchronzaton marker (the moton marker) defnes the boundar between the two components. The advantages of ths approach are twofold. Frst, an error wthn the teture data does not effect the decodng of shape. Second, data parttonng facltates unequal error protecton, whch can be emploed to ncrease the protecton of transmtted moton and shape nformaton from channel errors. 3) Inserton of a vdeo packet header. Ths header can appear perodcall n the bt stream. When present, t servers as a resnchronzaton marker and denotes the start of a MB. Addtonall, the header contans nformaton for decodng the transmtted MBs, even f prevous MBs were lost n the channel. Varous approaches to conceal shape nformaton have been proposed. However, most of them are based on a moton compensaton scheme. These concealment methods are lmted b the ntrnsc setbacks of moton compensaton, such as error propagaton, and perform poorl when objects appear/dsappear, rotate or are dstorted. Spatal concealment methods b nature bpass these problems. A MAP estmator wth an MRF desgned for bnar shape nformaton of neghborng blocks was proposed n [5]. It uses the spatal redundanc n transmtted shape nformaton on a statstcal base, and has not full tapped nto correlaton and dependenc between shapes of the neghborng areas. In [6] a more geometrc pont of vew s taken and the bnar alpha plane s nterpreted as a closed boundar that s mssng some peces. Ths s the same nterpretaton we use n ths paper and n [4]. In [5] the mssng boundar peces were then estmated usng a second order Bezer curve that matched the estmated dervatves at both connectng ends. The major problem wth ths technque s that second order Bezer curves are conve, though there s no reason wh the lost boundar pece must be conve. In our tests ths convet constrant resulted often n ncorrect appromatons.. PROPOSED CURVE-FITIG ALGORITHM The frst algorthm for shape concealment, whch we call the curve-fttng algorthm, s descrbed n ths

2 secton. ote that through the entre paper t s assumed that the alpha plane s n the bnar mode and the mssng area s of dmenson 66 pels, correspondng to one MB. These are the same assumptons as n [5]... Detectng Contet Frst, the mssng MB s detected from the receved vdeo packets. The four rectangular neghborng MB s are read and recorded as north, south, west and east (as shown n Fgure ). In order to take the dagonal neghbors nto consderaton, each block etends 4 pels on the two sdes (as shown n Fgure ) and therefore becomes 46. A coordnate sstem s set for south as shown n Fgure 3. orth, west and east are rotated to the poston of south so as to use the same set of coordnates and concealment functons. et, lnes that go over the -as are detected for each of the four neghbors: Frst, the 66 neghbor matr s scanned row b row, and boundar ponts (-> or ->) n each row are located. Then lne fttng s used on these ponts to generate a number of lnes equal to the number of boundar ponts n the border row, descrbed b two parameters: k (slope) and (crossng poston on -as). See Fgure 3 for an eample... Parng Up Lnes... Parng Scheme Canddates Snce the contour of a VO s a closed curve, each lne that goes nto a block must come out of t,.e., L all = Ln + Ls + Lw + Le ( Ln s the number of lnes crossng the north border, etc.) must be an even number. The L lnes are dvded nto L / pars so that each all par of lnes s gong to jon up n the MB to be concealed. Let Φ be an ordered (counterclockwse) table of all boundar lnes,.e. Φ = { ls, ls,... lsls, le, le,... lele, ln, ln,... lnln, lw, lw,..., l, Lw l wlw}, w par... all parl all / where l s and lsls are the westmost and eastmost lnes respectvel from the south border. Lkewse, let Φ be the clockwse table of the same set of lnes,.e. Φ = l, l,... l, l,..., l, l,..., l, l,..., l l { } 3 s wlw w nln n ele e sls s 3 s {... par par Lall / There are more possble parng schemes, but n ths paper, onl the most smple two are consdered (see Fgure 4 for an eample). In the future we wll show a general scheme of fndng the optmal parng, but that scheme needs to take nto account global knowledge and would eceed the scope of the current paper.... Estmatng the Mssng Curves Under the assumpton of the counterclockwse scheme, each par of lnes l and l have ponts (, ), =,, L from the contet detecton. If we assume a second-order curve model for the mssng curve that jons l and l up, then we have for have a + b + c + d + e = () (, ), =,, L. Pluggng these ponts n, we A a b c = M d e p = u L () The curve parameters are then estmated n a MSE sense: T T p = ( A A) A u (3) p = a b wth [ ] T c d e. ote that n the above formulaton we do not force that the resultng curve goes through the ntersectng pont of l and l wth the mssng macro block. Clearl we can change the above formulaton n that we epress two of the fve free parameters n terms of the other three such that the resultng second order curve must pass through the ntersectng ponts. Furthermore we are not usng the orgnal ponts of the boundar to estmate ths curve, but ponts from the estmated lnes. We could use the orgnal boundar ponts. Our eperments have shown that forcng the curve to go through the ntersectng ponts and usng the orgnal ponts results n a less robust algorthm..3. Makng a Decson Between the Canddates Detect the tangent slope of the concealed curve at ever boundar pont, and record t as ' k, =,...,. Calculate the sum of squares of the L all modfcatons for all lnes n Φ, denoted b ' = k k ) Φ l ( (4) Repeat.. under the clockwse parng scheme. Calculate the sum of square of the new modfcatons for all. lnes, denoted b B defnton of, we can see that the larger s, the greater the tangent slopes at the boundar ponts change before and after the concealment. amel, measures the overall smoothness of the concealed shape. As a result, we adopt the parng scheme (as well as the correspondng concealed curves) under whch (=,) s the smallest. Agan, n the future, we wll present a globall optmal wa of selectng the local parng scheme,

3 though ths local scheme works well n man practcal cases..3. Fll n the Mssng Area wth Bnar Colors Convert the concealed curve n the mssng block nto a 66 bnar matr. Bnar color nformaton has been passed on wth the lne nformaton throughout the prevous steps, so that t can be recovered. 3. PROPOSED HERMITE SPLIE ALGORITHM In ths secton, the second algorthm for shape concealment, called the Hermte splne algorthm, s descrbed. The development of ths algorthm has been nspred b the problems we encountered wth the curve fttng algorthm presented n the prevous secton. Whle the mplct form of the second order curve allows for an eas formulaton of the MSE curve parameters, t makes t hard to draw t effcentl. Furthermore, t also requres that we estmate the slope (d/d) of the ntersecton boundar ponts, whch becomes cumbersome when the slopes go towards +/-nfnt. One has to consder these specal cases carefull. In the above secton we have solved ths problem b usng the changng coordnate sstem. As t turns out, the Bezer based method presented n [6] has the same problem, snce t also requres the estmaton of d/d though no soluton has been presented n [6]. Furthermore, the Bezer approach and the curve fttng approach (when a second order curve s used) mpl that the estmated boundar pece must be conve. To solve the above problems we propose usng a cubc Hermte splne, whch s a parametrc curve defned as follows: p s) = p a = a + b 3 p 3 + b + c + c + d (. And ts gradent vector s therefore: dp( s) a = dp ds 3 ( s) 3a ds + b + b + c + c + d Clearl b usng a parametrc curve to nterpolate the boundar, the nterpolaton scheme s smple and effcent. In addton to ths, b usng a cubc Hermte splne, we have 4 parameters each n the and dmensons we can estmate. As can be seen n the above formulaton, the splne can match two ponts and the gradents at those ponts eactl. In other words, we can estmate the 8 parameters usng two ntersecton ponts and the approprate gradents at those ponts. A cubc splne s clearl the lowest order splne that can do that and we want to use as low of an order as possble to avod a wggl nterpolaton. So the problem at hand s now twofold, frst, how do we estmate the gradents at the ntersecton ponts of the orgnal boundar and the mssng macro block and second, how do we use that nformaton to estmate the parameters of the splne so t can be drawn. 3.. Estmaton of the gradents As dscussed n the prevous secton, we need the two ntersecton ponts and the gradents at those ponts. From our preprocessng steps we know the ntersecton ponts and the part of the boundar that runs nto these connecton ponts. Snce we need to estmate a gradent along a parameter s, and then want to match that gradent for drawng another parametrc curve, t s mportant that the parameter s has a unt that s meanngful. That means that we cannot use the common s method for drawng or estmatng the splnes. If we would, matchng the gradents along s from a short boundar pece to a long nterpolaton pece would be meanngless. We suggest usng the appromated path length n pels as the unt of s. Ths wll result n gradents that are meanngful no matter how short the boundar pece s versus how long the nterpolaton s. Wth ths n mnd, we propose usng a second order Hermte splne whch s forced through the ntersecton pont and appromates the known boundar as well as possble n the MSE, for estmatng the gradents at the ntersecton pont. q = q e = e q + f + f + g + g Clearl we could use a frst order Hermte splne (a lne), though that would not result n a good appromaton of the boundar pece f the boundar s bent even a lttle bt. A hgher order curve s also possble but the hgher the order the more data s needed for a good estmate, and sometmes there are ver few boundar ponts avalable. Assume that the ntersecton pont s (, ) then we force q(s) to go through ths pont at s =, hence g = and g =. The other parameters can be estmated usng the other ponts on the boundar pece connected to the ntersecton pont (, ) usng the followng relatonshp and then the generalzed nverse. s s s3 M s s s e s 3 f M s = 3 M In the above formulaton onl the coordnates are shown, clearl for the coordnates a smlar equaton holds. It s mportant to enter the correct values for the parameter s. As mentoned before, we use the appromate path length

4 n pels, whch means snce s s a 4-connect neghbor of s (whch s ) t s ether or.4. The other s parameters can be appromated n the same wa. The result of the above MSE estmaton of the parameters of the second order Hermte splne forced through (, ) s that we have an estmate for the gradents at (, ) whch s smpl c c. We now do ths for ever ntersecton pont and collect all these estmates of the gradents that wll be used later on to estmate the cubc Hermte splne nterpolaton. 3.. Estmaton of the cubc Hermt splne In the followng secton we assume that we have alread found the correct match between ntersecton ponts. In ths secton, we smpl use the same as found n the curve fttng method. As alread mentoned, there s a more general soluton to the problem that though needs to take nto account the entre boundar. In ths paper, we smpl consder local decsons. We have now two ntersecton ponts and ther respectve gradent estmates along a parameter s, whch has a unt of pel along the curve. Snce we defned n the prevous secton that the parameter s starts at the ntersecton pont and appromates the boundar movng awa from the macro block as s ncreases, we need to change ths defnton for one of the ntersecton ponts, so that all parameters, that of the frst boundar appromaton, that of the boundar nterpolaton and that of the second boundar appromaton move nto the same drecton. Currentl the frst and the second boundar appromaton splnes are movng n opposte drectons. Ths s easl dealt wth b changng the sgn of the frst one of the gradents whch then results n a gradent belongng to a varable s that moves n the correct drecton. Havng done ths, we can now solve the followng problem for the four unknown a, b, c and d : sm 3sm a b c d () ( s m = d s= d ds s= ds ) 3 sm sm s m s m Because of the smple structure of ths set of lnear equatons, the soluton can be found easl. ote that s m s the value of the s parameter used for the nterpolaton splne. It starts at zero and moves up to ts mamum, s m. We estmate s m as the dstance between the two ntersecton ponts. That gves us an parameter s that s appromatel of the correct length and hence matchng the gradents along the dfferent parameters makes sense and leads to ver good results. Agan, we onl showed the equaton for the coordnate, an equvalent equaton for the coordnate needs to be solved also Fll n the Mssng Area wth Bnar Colors In the Hermt splne method, ths s acheved b connectng the receved boundar and the nterpolaton splnes to one closed boundar. We then use a smple recursve fll algorthm to set the outsde of the boundar (and the boundar) to the background, and everthng else to the foreground (the object) 4. SIMULATIO RESULTS The performance of the two proposed concealment algorthms s measured b the rato of the number of correctl concealed pels over the number of mssng pels. Table shows the performance of the proposed concealment sstems ( Curve Fttng and Hermte Splne ), as well as, that of two other estng ones ( MAP [5] and Medan ). 3 realzatons have been carred out for each error rate. See Fgure 5 to 8 for comparsons of vsual concealed results. Error Rate Hermte Splne Curve Fttng MAP Medan % 98.7% 95.5% 9.% 4.% 8% 97.% 95.8% 9.6% 9.5% 6% 96.5% 95.6% 9.6% 88.8% 4% 96.% 94.8% 9.4% 89.4% Average 97.% 95.4% 9.4% 77.7% Table. Shape smlart measure for Bream frame 5. COCLUSIOS The proposed shape concealment methods nterpret shape nformaton n terms of boundar lnes. The eplore the known shape nformaton on a relatvel hgh level such as slopes and curvatures, and thus makng better use of the avalable nformaton then the statstcal methods. For eample, the Hermte splne method acheves n our tests a 9.4% and 4.7% mprovement over the tradtonal medan method and the recent MAP method [5], respectvel. What s even more mportant s the mprovement n subjectve qualt, whch s hard to measure, but the mages provded n fgure 5 through 8 show that both curve-based methods are superor to the statstcal methods. Moreover both proposed technque consumes far less computaton resource than the MAP method and are capable of beng mplemented n a real tme communcaton sstem. The net etenson of ths work s an mproved parng scheme, whch keeps all nonntersectng combnatons of possble pars per concealment regon. It then checks whch of the possble combnatons would result n one closed, non-ntersectng boundar and select that one as the optmal one.

5 6. REFERECES [] Tet for ISO/IEC FDIS Vsual, ISO/IEC JTC/ SC9/ WG 5, ov [] A. Pur, et al., Multmeda Sstems, Standards, and etworks, Marcel Dekker, Inc, 999 [3] Y. Wang, et al., Error Reslent Vdeo Codng Technques Real-Tme Vdeo Communcaton Over Unrelable etworks, IEEE Sgnal Processng Magazne, Jul. [4] X. L, A. K. Katsaggelos, G. Schuster, A Recursve Shape Concealment Algorthm, ICIP. [5] S. Shran, et al., A Concealment Method for Shape Informaton n MPEG-4 Coded Vdeo Sequences, IEEE Trans. on Multmeda, Sept. [6] Me-Juan Chen, et al., Spatal and temporal error concealment algorthms of shape nformaton for MPEG-4 vdeo, ICCE Fgure 3. Ponts detected from southern border. 7. FIGURES Fgure. Alpha plane of frame of bream, VOP boundar s shown n whte. Fgure 4. Two canddate-parng schemes. Fgure. orth, east, south, west neghborng MB s used for the concealment of the MB at the center. Fgure 5. () Corrupted VOP wth 3.7% packets loss; () Concealed VOP from medan method, concealng

6 rate=85.49%; (3) Concealed VOP from MAP method, concealng rate=95.7%; (4) Concealed VOP from curve fttng method, concealng rate=98.63% () Fgure 6. () Corrupted VOP wth.7% packets loss; () Concealed VOP from medan method, concealng rate=9.6%; (3) Concealed VOP from MAP method, concealng rate=9.3%; (4) Concealed VOP from curve fttng method, concealng rate=97.% () (3) Fgure 7. () Corrupted VOP wth.63% packets loss; () Concealed VOP from medan method, concealng rate=9.63%; (3) Concealed VOP from MAP method, concealng rate=94.5%; (4) Concealed VOP from curve fttng method, concealng rate=96.4%. Fgure 8. Hermte Splne Method: () Concealed VOP, 3.7% packet loss, concealng rate=99.%; () Concealed VOP,.7% packet loss, concealng rate=99.%; (3) Concealed VOP,.63% packet loss, concealng rate=98.7%.