Publshed on Pcture Codng Symposum 1999, March 1999 Effcent Vdeo Codng wth R-D Constraned Quadtree Segmentaton Cha-Wen Ln Computer and Communcaton Research Labs Industral Technology Research Insttute Hsnchu, Tawan 310, ROC ljw@n100.ccl.tr.org.tw Yao-Jen Chang, Eryn Fe, and Yung-Chang Chen Department of Electrcal Engneerng atonal Tsng Hua Unversty Hsnchu, Tawan 300, ROC {kc,eryn,ycchen}@benz.ee.nthu.edu.tw ABSTRACT In ths paper, a rate-dstorton framework s proposed to defne a jontly optmal dsplacement vector feld estmaton(dvfe) and quadtree segmentaton technque for vdeo codng. Ths technque acheves maxmum reconstructed mage qualty under the constrant of a target bt rate for the codng of the dsplacement vector feld, quadtree segmentaton nformaton, and the resdual sgnal. A fast herarchcal moton estmaton scheme as well as a face model-asssted condtonal optmzaton strategy s proposed to drastcally reduce the large computaton cost requred n the R-D optmzaton process. A novel skncolor based face detecton scheme s also proposed for fast locatng face regons. The smulaton results show that the proposed method can acheve more than 0.5 db PSR qualty mprovement over the H.263 TM5 vdeo codec at an acceptable extra computaton cost. 1. ITRODUCTIO Rate-dstorton optmzaton technque has recently been wdely nvestgated for the applcatons n mage and vdeo codng, because t can acheve maxmum reconstructed mage qualty under the constrant of a target bt rate [3-4]. The effect of ratedstorton optmzaton would be more obvous when dealng wth low bt rate vdeo codng, snce the rato of bts needed for encodng the dsplacement vector feld and segmentaton nformaton wll become relatvely large n such cases. Therefore, t s desrable for low bt-rate vdeo codng to reduce as much as possble the bt rate needed to transmt the dsplacement vector feld and segmentaton nformaton, provded that ths reducton does not produce ntolerable dstorton n the reconstructed mage. Quadtree segmentaton has been wdely adopted n varable block-sze vdeo codng to effectvely reduce the number of transmtted moton vectors as well as mantan the moton ntegrty n large unform regons whch are often segmented nto smaller blocks n fxed block-sze vdeo codng schemes. The gan of such knd of varable-block sze schemes over the tradtonal fxed block-sze schemes s especally obvous n the applcaton of low-moton vdeo (e.g., head-and-shoulder mages n vdeo phone applcatons). Combng quadtree segmentaton scheme wth R-D optmzaton concept can further mprove the codng performance as shown n [2]. The optmzaton of segmentaton n rate-dstorton sense s, however, very computaton ntensve thereby makng t mpractcal n real-tme applcatons. In ths paper, we develop a framework of R-D constraned quadtree segmentaton for vdeo codng. A fast herarchcal moton estmaton scheme as well as a model-asssted condtonal optmzaton strategy s proposed to drastcally reduce the huge computaton load requred for R-D optmzaton process. 2. R-D COSTRAIED VIDEO CODIG Let v V be the dsplacement vector correspondng to the block of the mage, where V s the set of all dsplacement vectors determned by the proposed search algorthm. The purpose of the R-D constraned vdeo codng s to mnmze the dstorton D of the reconstructed mage sequence, under the constrant of the target rate R target for transmttng the dsplacement vector feld, the segmentaton nformaton and the error mage. Ths corresponds to the followng constraned optmzaton problem: mn D ( v, s ) (1) v V subject to R ( v, s ) R target where s the total number of blocks n the mage, D(v,s ) s the contrbuton of the jontly consdered par (v,s ) to the overall dstorton, and R(v,s ) s the contrbuton of (v,s ) to the total rate. In the proposed R-D constraned codng method, the rate part s composed of three components: one s the bt rate for transmttng the dsplacement vector feld, another one s the bt rate for sendng the quadtree segmentaton nformaton, and the rest s for codng the error mage. On the other hand, the dstorton part s determned by means of Dsplaced Frame Dfference (DFD). From the methodology shown n [4-5], the above problem can be transformed nto an unconstraned optmzaton problem by adoptng the Lagrange multpler λ. Thus the soluton { (v * (λ), s * (λ)),,..., } of the unconstraned mnmzaton of the cost functon C(λ): C ( λ ) = C = + D ( λ ) λ R ( λ ) (2) = D [ v ( λ ), s ( λ )] + λ R[ v ( λ ), s ( λ )] (3) s also a soluton of (1) f: = * * R target R [ v ( λ ), s ( λ )] (4) It was shown n [3] that D( λ ) and R( λ ) are monotonc functons of the Lagrange multpler λ, wth values rangng from
zero (hghest rate, lowest dstorton) to (lowest rate, hghest dstorton). A value of λ corresponds to a (R, D) operatng pont. Snce the relatonshp between D(λ) and R(λ) s nearly one-to-one, all we have to do s to fnd an optmal Lagrange multpler λ * whch makes R( λ * ) close to R target. The correspondng soluton { (v * (λ), s * (λ)),,..., } consttutes the optmal dsplacement vector feld under the target rate constrant. 3. THE PROPOSED R-D COSTRAIED QUADTREE SEGMETATIO The proposed archtecture to realze the aforementoned framework of R-D constraned vdeo codng s depcted n Fgure 1. A herarchcal splttng scheme based on moton nformaton s used to segment an mage nto varable-sze blocks wth unform moton. The R-D optmzaton process performs quadtree segmentaton and moton estmaton n an teratve fashon to fnd the best match whch s too computaton ntensve to be used for real-tme vdeo communcaton. In order to reduce the computaton load, some strateges are adopted to speed up the optmzaton process wthout ntroducng severe degradaton. Frstly, a fast herarchcal estmaton scheme smlar to our prevously proposed approach n [1] s utlzed to effectvely reduce the computaton load n moton estmaton. Furthermore, n our proposed method, the rate-dstorton optmzaton s not appled to the whole mage, nstead t s only appled to the movng regons. Ths face model-asssted condtonal optmzaton strategy can also save much computaton tme whle stll mantanng good vdeo qualty, especally on the regons of specal nterest. 3.2 A ovel Skn-Color Based Face Detector As aforementoned, face regons are of specal nterest to be R-D optmzed n our work. Therefore an effcent detecton scheme s requred to locate face regons. It was shown n [5] that human skn-color based face detecton approach can effectvely dentfy the face locaton n real-tme. In ths work, we propose a novel low-complexty skn-color based face detecton scheme whch s robust aganst camera noses and skn-color-lke nterference of non-face objects by usng jont moton/color probablty model asssted decson. As shown n Fgure 2, each pxel on the nput vdeo frame s classfed as ether skn color or non-skn color va the pre-determned skn-color model. A bnary-tree splt-andmerge segmentaton process s performed to group the skn-color pxels nto canddate face regons. Subsequently, A jont moton/color probablty model s used to determne the most probable face regon wth the hghest jont probablty. After locatng the face regon of the frst frame, the fast face trackng mode s used to speed up the detecton process. Each functonal unt s elaborated below. A. The Moton Probablty Map for Each Pxel For vdeo sequences, the frame dfferences provde the moton nformaton. We have made an assumpton that most motons n the vdeo sequences are caused by objects n the head. The assumpton s reasonable for vdeo sequences manly contanng a person wth head and shoulders, such as, the news programs, the vdeophone and the vdeoconference vdeo, etc. Therefore, the locaton wth hgher the frame dfferences, the more probable the locaton s n the face regon. Actually, by our observaton, most motons are caused by the eyes blnkng, the mouth s openng and closng, and the head movements. We use the followng equaton to calculate the frame dfferences: Df (, f (, f (, (5) = n t= 0 t t 1 where the f t (, denotes the pxel value n the locaton (, of the vdeo sequence at tme t. And the f t-1 (, denotes the pxel value at tme t-1. The probablty for the pxel at locaton (, belongs to the face regon s calculated n Eq. (6) P face(, = Df (, Df ( k, l) (6) k l B. Color Probablty Map for Each Pxel The Bayesan decson rule descrbed n [5] s adopted to set up the color probablty model for each pxel n the Y-Cb-Cr color coordnate. In ths work, the threshold TH for skn color classfcaton s automatcally set under the assumpton that the face regon cannot occupy more than sxty percent area of the whole frame. In ths way, background nose can be reduced for detecton of the face block. A bnary search algorthm s used for fast auto-threshold settng. C. Bnary-Tree Splt-and-Merge Segmentaton After thresholdng by usng TH, there may stll exst some background nose or false detecton of other objects n the mage wth skn-lke colors. Therefore, further segmentaton process s requred to elmnate the nose regon and separate dfferent objects. Here, we propose a bnary-doman splt-and-merge algorthm wth bnary-tree parttonng. eghborng pxels are grouped to form a face block canddate. Snce the bnary-tree segmentaton s performed n the bnary doman, the computaton cost s very low. The splt-and-merge algorthm s summarzed as follows: Splt Phase: 1. The search regon s set to be the whole pcture. 2. Fnd the upper and lower boundares of the skn-color block by thresholdng the horzontal ntegral projecton of the search regon. 3. Fnd the left and rght boundares of the skn-color block by thresholdng the vertcal ntegral projecton of the pcture between the upper and lower boundares. 4. If the fullness of the skn-color block s hgher than 60% or the depth of the splt s hgher than the predefned maxmum depth (max depth = 3 n the proposed scheme), goto step 5. Otherwse, dvde the block equally n the horzontal drecton to two search regons and repeat steps 2~4 for each of the two search regons wth an ncreased depth. 5. Stop the splt process for the current block and record the block boundary nformaton. Merge Phase: For each recorded block resultng from the splt phase, merge those connected blocks that have small dfferences n block
wdth to form a skn-color group. And the wdth of the group s the weghted averages of the blocks belong to the group. D. Jont Moton/Color Probablty Model Asssted Decson The normalzed moton probablty for each group s defned as follows: 1 Pface (, (7) Area ( Group n ) (, j ) Group n Pm ( Group n ) = # of Groups 1 Pface (, Area ( Group ) k = 1 k (, Group k And the normalzed fullness of skn-color for each group s defned as : rato of skn - color pxels n Group (8) n F( Group n ) = # of Groups rato of skn - color pxels n Group k= 1 Wth these two knds of nformaton for each group, we set dfferent weghtng factors W m and W f for Pm(.) and F(.) respectvely. The probablty for a group to be the face block can be calculated by Eq. (9): Pfb( Groupn) = Wm Pm ( Groupn) + W f F(Groupn ) (9) The group wth maxmum value of P fb s determned as the face block. The weghtng factors W m and W f are emprcally set as 1/3 and 2/3 respectvely. E. Fast Face Trackng Mode After we extract the average color value n the detected group, unlke the general skn-color dstrbuton that covers moderate ranges n the Cb-Cr plane, the skn-color for a specfc person s restrcted to a much smaller range. Hence, the extracted average color of the face regon n the prevous frame can be used as a very relable reference model for trackng the face n the current frame. Meanwhle, the detected locaton of prevous frame s taken as the ntal guess to search the current face locaton wthn a small search wndow to speed up the trackng process. 3.2 Model-Asssted Vdeo Codng wth R-D Constraned Quadtree Segmentaton To save computaton, the R-D optmzaton process s performed only on those regons of specal nterest. In vdeo phone and vdeoconference applcatons, face regons are the most mportant parts for human percepton. As shown n Fgure 1, a change detector s used to detect the movng regons and the statc ones wthn the face regon. Frstly, the dfference mage between the faces n the current mage and the prevous mage s determned. If an absolute pxel value of the dfference mage s lower than a threshold, the pxel s classfed as statc, otherwse t s classfed as movng. If over ffty percent of pxels n a regon are movng, then the regon s regarded as a movng regon, otherwse t s a statc regon. Meanwhle, postprocessng s performed to fll the holes and elmnate the solated regons. The quadtree segmentaton s performed by splttng blocks of a predefned large block sze nto smaller blocks wth unform moton n a top-down manner. Before processng the splt phase, a splt crteron should be determned. That s, f the cost of splttng a block s smaller than no splttng, then the block s splt, otherwse, not splt. For the purpose of rate-dstorton k optmzaton, the cost functon s the summaton of rate and dstorton wth the Lagrange multpler λ as the weghtng coeffcent as defned n (3). As mentoned above, only those blocks whch are classfed as movng blocks wthn face regon are quadtree segmented. 4. EXPERIMETAL RESULTS Table 1 compares the smulaton results of average PSR, bts/pxel and relatve computaton tme wth the test mage sequence Clare and Mss Amerca encoded at 384 kbts/sec usng the proposed jontly R-D constraned quadtree segmentaton and moton estmaton method and the tradtonal TM5 H.263 coder respectvely. The expermental results show that the proposed scheme can acheve more than 0.5 db PSR mprovement over the TM5 coder. The computaton cost requred for the proposed scheme s, however, a bt hgher than the TM5 coder. Table 1 also ndcates the average number of bts requred for encodng the moton vectors, the segmentaton nformaton, the predcton errors, and I frames. It s shown that the number of bts requred for encodng the moton vectors s consderably reduced such that more data bts can be assgned to encode the resduals thus leadng to performance mprovement. Ths codng strategy wll be especally advantageous n very low bt rate applcatons. Fgure 3 llustrates the quadtree segmentaton results at each stage. Only about 15% regons are classfed as movng regons as shown n Fgure 3, thus the R-D computaton only needs to be performed on a small porton of an mage wth the proposed method. The proposed method can also be combned wth some model-asssted rate control schemes to further emphasze the qualty on the face regons [6] 5. REFERECES [1] Cha-Wen Ln, Eryn Fe, and Yung-Chang Chen "Herarchcal dsparty estmaton usng spatal correlaton, IEEE Trans. Consumer Electroncs, vol. 44, no. 3, pp. 630-637, Aug. 1998. [2] G. J. Sullvan and R. L. Baker, "Effcent quadtree codng of mage and vdeo," IEEE Trans. Image Processng, vol. 3, no. 4, pp. 327-331, May 1994. [3] Y. Shoham and A. Gersho, "Effcent bt allocaton for an arbtrary set of quantzers," IEEE Trans. Acoustc, Speech, and Sgnal Proc., vol. 36, pp. 1445-1453, Sept. 1988. [4] K. Ramchandran and M. Vetterl, "Best wavelet packet bases n a rate-dstorton sense," IEEE Trans. Sgnal Proc., vol. 2, pp. 160-175, Apr. 1993. [5] Hualu Wang, and Shuh-Fu Chang, A Hghly Effcent System for Automatc Face Regon Detecton n MPEG Vdeo, IEEE Trans. on Crcuts and Systems for Vdeo Technology, Vol.7, o. 4, 1997. [6] J.-B. Lee and A. Eleftherads, "Spato-Temporal Model- Asssted Compatble Codng for Low and Very Low Btrate Vdeotelephony", ICIP-96, Lausanne, Swtzerland, Sept. 1996, pp. II.429-II.432.
I'(t) Memory Buffer + IDCT Q -1 Q DCT + + - Recon structed I L ''(t) Moton Compensaton Predcton Errors I(t) I'(t-1) Face Mode- Asssted Change Detector Movng Regons Statc Regons HBM Moton Estmaton Splt Segmentaton Moton Vectors Segmentaton Informaton Fgure 1. The proposed R-D contraned vdeo encoder (a) Vdeo Sequences Frame Dfferences Moton Informaton Moton Probablty Map for Each Pxel P face (, Moton Probablty Map for Each Group Pm(Group n ) Weghtng : Wm Extract the Face Color Trackng The Face Block max n (P fb (Group n )) Vdeo Sequences The Frst Frame The Color Components of the Frame Color Probablty Map for Each Pxel P(pxel value skn - color) Segmentaton Possble Face Blocks F(Group n ) = fullness of skncolor n the group Weghtng : Wf (b) The Bayesan Decson Rule Skn-color Model Fgure 2. The flow chart of the propose skn-color based face detecton scheme H.263 (TM5) Proposed Scheme H.263 (TM5) Proposed Scheme Clar (384 Kb/s) PSR (db) bts/pxel Tme 36.51 0.0885(M:0.0026, E:0.0355, I:0.0504) 37.05 0.0881(M:0.0014, S:0.0004, E:0.0361, I:0.0504) 100% 114.3% (a) Mss Amerca (384 Kb/s) PSR (db) bts/pxel Tme 38.03 0.0479(M:0.0051, E:0.0217, I:0.0211) 38.61 0.0482(M:0.0021, S:0.0014, E:0.0236, I:0.0211) 100% 145.6% (b) Table 1. Performance of the proposed stereoscopc codng scheme and two other methods. M, S, E, and I n the tem of bts/pxel denote the percentage of the number of bts needed for codng moton vectors, segmentaton nformaton, predcton errors, and the ntra frame respectvely. (c) (d) (e) (f)
Fgure 3. (a) orgnal mage of frame 0, (b) face detecton result of (a), (c) ntal segmentaton result, (d) the ndcaton of movng objects n (c) (whte areas), (e ), (f) the fnal segmentaton result.