2. KEYFRAME CONSTRUCTION

Transcription

1 Proceedngs of the Fourth IIEEJ Internatonal Worshop on Iage Electroncs and Vsual oputng oh Sau, Thaland, October 7-0, 04 SPETRAL-ASED VIDEO OJET SEGMENTATION USING ALPHA MATTING AND AGROUND SUTRATION a Rur Suo asu, b Moch. Aref Soelean, c Mochaad Harad, d Maurdh Hery Purnoo e Rcardus Angg Praunendar, f Aura Faranta Yoganant, a,b,c,d Faculty of Industral Technology, Dept. of Electrcal Engneerng, Insttut Tenolog Sepuluh Nopeber, Surabaya, Indonesa a,b,e,f Faculty of oputer Scence, Dan Nuswantoro Unversty, Searang, Indonesa E-al: rur.basu0@hs.ee.ts.ac.d, rursb@research.dnus.ac.d ASTRAT The an objectve of ths study s to separate object fro vdeo sequences. To separate the object fro the vdeo data s perfored by cobne several algorths. The frst stages, the vdeo data s splt nto several fraes, and the ntal frae s treated as a eyfrae. Object extracton on the eyfrae requre huan nterventon, naely by gvng scrbble on the regons of foreground and bacground. Mattng technque uses a closed-for soluton ethod appled n ths stage. Further, the results used as a reference for object extracton n subsequent fraes. To get the labels on the next fraes, bacground subtracton algorth s appled, and the result s used as the nput age on the next fraes. So that the object extracton n subsequent fraes can be perfored repeatedly usng attng technques. The experental results show that the object extracton at the ntal frae shows good results. However, the accuracy decreases when the object oves too fast and suddenly. eyword : Segentaton, Alpha Mattng, acground Subracton. INTRODUTION The advent of dgtal vdeo standards such as Dgtal Vdeo roadcastng (DV), Dgtal Vdeo roadcastng - Terrestral (DV - T) and Integrated Servces Dgtal roadcastng - Terrestral (ISD - T) s pushng the deand of the vdeo edtng applcatons (such as vdeo segentaton and vdeo copostng) and rapdly ncreased, snce t plays an portant role n the producton of oves, news and advertsng. The object-based technology can be used n varous applcatons, such as object extracton, oton understandng, age recognton and augented realty. Unfortunately, the process of object segentaton of vdeo becoes a dffcult job, snce the vdeo object has no seantc nforaton. In other words, a vdeo object segentaton s an llposed proble []. Therefore, the pullng of objects n vdeo edtng s perfored wth anual segentaton, snce the seantc object can only be dentfed by huans vson that consders the vdeo context. However, ths ethod s not effectve when t s appled to the vdeo data wth large volues. To overcoe ths proble, several algorths related to vdeo object segentaton have been developed. Generally, these algorths are classfed nto two categores, naely the autoatc object segentaton [] and a se-autoatc object segentaton [3] [4]. Autoatc segentaton s done wthout huan nterventon by consderng specfc characterstcs such as color, texture and oveent. The an proble of the autoatc segentaton s the dffculty n objects separatng whch s seantcally eanngful. Untl today, there s no guarantee that the result of the autoatc object segentaton wll be satsfactory, snce the seantcally object has a lot of color, texture and oveent [5] [6] [7]. For ths reason, several se-autoatc segentaton ethods are proposed as a cobnaton of the autoatc segentaton and anual segentaton. In prncple, ths approach s a technque to pull the object that nvolves user nterventon at several stages of the segentaton process. Thus seantc nforaton can be defned drectly by the user. Furtherore, object segentaton n subsequent fraes s perfored usng a tracng echans by teporal transforaton. Soe of ethods used for tracng echanss has been appled n several prevous studes. In a regon-based ethod, the related areas are n accordance wth the shape of seantc objects traced by the oton, texture and color paraeters [6] [7]. Weanesses of the ethod are very coplex tracng echans n antanng relatonshps between regons coposed of seantc object [8]. Meanwhle, the contour-based ethods, such as snaes [3] wll robust when trac the object contours nstead the whole of the object that coprsng the pxels, so that these ethods ay not

2 wor well when the feature to be followed naely edges are not connected to each other. Whereas the odel-based ethod apply a pror nowledge of the object shape. The drawbac of ths approach s not acceptable on the generc seantc vdeo object segentaton due to the detal needs nforaton about the object geoetry [9]. In ths paper, se-autoatc vdeo segentaton fraewor s proposed to be appled to the general vdeo data. Early stages n vdeo segentaton s perfored by creatng a "eyfrae" whch s used as a reference for tracng process on the subsequent fraes. Hereafter, the object segentaton on the subsequent fraes s done by usng the bacground substracton algorth.. EYFRAME ONSTRUTION The frst stage of a vdeo segentaton process s done by constructng the ntal frae of the sequence scene whch becoes a ey frae. Snce the object have no the seantc nforaton, scrbble s used as a label to dstngush areas that represent foreground and bacground (whte color for foreground and blac color for bacground). Next, the object s extracted wth attng technques. A. General opostng Equaton Alpha channel [0][][] s used to control the lnear nterpolaton n the foreground and bacground whch are depcted n attng algorth by assung that each pxel n the nput age I s a lnear cobnaton of the color of foreground F and bacground. I F, where 0 () based copostng equaton Eq.() of each pxel s assued to be a convex cobnaton of layers age whch denoted as I F () the fractonal contrbuton of each layer observed n each pxel s deterned by the vector whch s a coponent of age attng.. Spectral Analyss of Spectral segentaton ethod s assocated wth the affnty atrx. For exaple, the age A, sze N x d j / N s assued as A(, j) e and d. In whch j d j s the space aong pxels (e.g. color and geodesc space), whch s defned as L D A (3) whle D s atrx degree fro graph. wth dagonal atrx where d G V, E wthv n (4), j D A,, j, j j deg v f j (5) 0 Otherwse D s stuffed wth degree nforaton of each (, j) vertex (node) wth D for G as rectangular atrx sze n x n depcted. So s a syetrc postve se-defnte atrx wth egenvector whch s able to capture a lot of age structure. Furtherore, the age s coposed of soe dfferent clusters or connected coponents whch can be captured by the affnty atrx. Subset n age pxel s the connected coponent of age for each (, j) so A and L j A (, j) 0, so there s no subset that can fulfll ths property. s defned as ndcator vector of coponent, therefore 0 (6) wth the assupton that the age conssts of connected coponents of,, to {,, N} = = wth dsjont path on the pxel, then the represents 0-egenvector (egenvector wth egenvalue 0) fro L. Snce the rotaton of atrx n sze, and vector [,, ] R s the null space base on L, then R, the ndcator vector,, resulted fro egenvector calculaton on s only reachng the rotaton. "Spectral Roundng" whch s a coponent extracton wth the sallest egenvector becoes a concern n soe studes [3] [4] [5] [6] [7]. -Means algorth s a sple approach used for clusterng the age pxels [3], whle the perturbaton analyss algorth s to lt errors as a functon of connectvty wthn and across clusters. ) Mattng Laplacan In order to evaluate the qualty atte wthout consderng colors of foreground and bacground Mattng Laplacan [0] s appled by usng a local wndow w forng two dfferent pathways n the RG doan as denoted n Eq.(6). Furtherore, α n w s expressed as a lnear cobnaton of color channels. L

3 w a I a I a I b (7) R R G G The devaton of lnear odel Eq.(7) n all the age wndow wq becoes one of the fndngs n a atte extracton probles. R R aq I G G (8) q q I w a q q I aq I b q J, a, b a the requreents whch ust be fulflled of the alpha s ε a q whch s a lnear odel coeffcents α,b that allows elnaton fro Eq.(8) and the result s a quadratc cost n α J T L, (9) It has the ordnary nu cost whch s a constant α vector, then n fraewor user-asssted [], J(α) s the subject nzed n user constrant. The equaton L (9) s attng Laplacan. Syetrc sedefnte postve atrx N x N s the atrx nsertng nput age functon n local wndows, whch depends on unnown foreground and bacground color n the coeffcent of lnear odel. L s defned by the su of atrx L = q A q n whch on each s flled wth affnty aong pxels n local wndow w q T j I q I 3x3 I j q, A, j w q q q w q 0 Otherwse j where j, wq (0) q In whch s ronecer delta, s the average color vector n al pxel q, q s atrx covarant sze 3 3 n the sae wndows, w q s the su of pxels I 3 n wndow, and s dentty atrx sze 3 3. y the occurrence of the sallest egenvector, the other use of attng Laplacan property Eq.(0) s to catch nforaton of job fuzzy cluster on age pxel, ncludng the calculaton before the lt deterent by user s specfed [5]. ) Lnear Transforaton The lnear transforatons trac n egenvector wll produce a set of vector whch the value s adjacent to a bnary. The equaton denoted as E = [e,, e ] s converted to atrx N x of egenvector. Next to locate a set of lnear cobnaton, vector y nzes,, where Ey subject to () The robust easureent value n attng coponent [] s deterned by α γ + α γ, If 0 < γ <, thus, the value of γ = 0,9. ecause the cost functon Eq.() s not convex, the ntalzaton process deterne the result of Newton process. Therefore, -eans algorth can be used n the ntalzaton process on the sallest egenvector n attng Laplacan and projects ndcator vector of cluster resulted fro egenvector E. () EE T The attng coponent result Eq.() s then added. Thus t gves soluton for Eq.(). 3) Groupng oponent The coplete results of atte extracton of the foreground object are deterned by a sple suaton on the foreground. For exaple,,, n s desgned as a coponent of the foreground, so that n (3) The easureent of the results - atte s perfor by when the sallest egenvector s not equal to zero, n whch L s the attng Laplacan. The frst calculaton of correlaton aong attng coponent and L devaton n atrx s defned as T L then, atte cost s calculated as T l, l L (4) T J b b (5) where b s the bnner vector of -densonal ndcatng the chosen coponent.. 3. TRAING MEHANISM A. acground Subtracton acground subtracton [8] s used to dentfy dfferences n the ntensty of the current age wth the bacground. Frae dfference s the technque used n the bacground subtracton whch s a nonrecursve technques. Ths odel assued as F whch s bnner value of a foreground age. I x y n, f I x, y, n,, F( x, y, n) 0, otherwse (6) The threshold (α) used to classfy the foreground and bacground. Here, Otsu algorth used to generate the threshold value.. Otsu Threshold Otsu [9] s an adaptve threshold algorth based on the hstogra that shows the value of changes n ntensty of each pxel n one-densonal age. The

4 x-axs s used to express the dfference of ntensty levels, whle the y-axs s used to declare the nuber of pxels that have ntensty values. y usng the hstogra clusterng, the age pxel based on the threshold value can be done. Optal threshold s obtaned fro ntensty dfferences of the pxels, so that t can be used for separatng groups. The nforaton obtaned fro the hstogra s the aount of the ntensty dfference (denoted by L), and the nuber of pxels for each ntensty level s denoted by, wth ). Seeng of n the threshold value n Otsu algorth s perfored as follows:. alculate the hstogra of the noralzed age denoted by wth 0,,... L where n ntensty, and startng fro p p n MN ( 7) s the nuber of the pxels at each n 0 MN to s the nuber of n. opute the cuulatve nuber of for 0,,..., L. L P ( ) p 0 n p ( ) (8) 3. ount the coulatve average of ( ) for 0,,..., L. whle, L ( G ) p 0 Note : the value of s obtaned when axu. 4. EXPERIMENT AND EVALUATION ( ) (3) In ths experent, we evaluate our proposed algorths usng standard test vdeo sequences obtaned fro the UF Sports Acton Data Set (.e. rdng horse, lftng, sateboardng and forean) 30 fraes respectvely. Intal stages, the frst frae of the vdeo sequence s consdered as a stll age (shown n Fg. (a)). In our experents, the selected frae as a eyfrae s a frae that has ntact object of the entre vdeo sequence. Se-autoatc technque s perfored by gvng scrbble (as a label) to dstngush between areas of foreground and bacground (llustrated n Fg. (b)). Scrbble age uses bacground brush (blac scrbble n our exaples) to show the bacground pxels (α = 0) and foreground brush (whte scrbble) to show foreground pxels (α = ). In order to separate the foreground object fro the whole age, a attng technque [0][] s appled such as depcted n Fg. (c). s ( ) p (9) 0 4. alculate the average global ntensty usng ; G by g L p (0) 0 5. opute the varance between classes, ( ) for 0,,..., L. (a) ( ) P ( ) ( ) G P ( ) P ( ) () 6. Select a threshold value of the * f the ndex value of the axu varance between classes ( ax( )), and f the ndex value ore than one value of *, then the threshold value s deterned fro the average value of *. 7. Deterne the sze of the separaton * wth * ( ) ( ) () (b) (c)

5 Fg.. (a). Stll age, (b). Scrbble age, (c). Segentaton result Furtherore, to extract object on the subsequent fraes, we used bacground subtracton technque Eq.(6) to get dfference of the bnary value between current frae and prevous frae. nary value of s assued as label for foreground and 0 for the bacground. Ths value s then used to replace the role of scrbble and used n the process of attng n subsequent fraes. The exaple of segeentaton results of the vdeo data s shown n (fgure 3). To easure the accuracy of object segentaton, we evaluate usng the Mean Square Error (MSE) are denoted as follows: MSE j Grd. Truth Seg. Obj (, j) (, j) MN (4) Grd. Truth s the ground truth age resulted fro anual segentaton. Whereas Seg. Obj s the object that produced by the segentaton process. In ths experent, the MSE calculatons perfored around the fraes of the vdeo data test. The results are descrbed n (Fg. ). # #5 #0 # #5 #0 #5 #30 Fgure 3. Object segented References [] A. ovc, The Hand oo of Iage and Vdeo Processng, Acadec Press, 998. Fgure. MSE of Frae Sequences 5. ONLUSION AND FUTURE PLAN In ths paper, we proposed an approach to segent vdeo object se-autoatcally. Fro our experents on the 4 vdeo datasets each 30 frae, the lftng vdeo data ndcate that segentaton accuracy of the tracng s the ost stable, snce consst of ost delcate object oton. Whle the "forean" vdeo data, segentaton accuracy of the tracng loos rough on soe fraes, because there are objects that ove faster and all of a sudden. For future wor, In order to prove the accuracy of segentaton n subsequent studes, the ntensty value of vdeo data are classfed frst before tracng. [] H. Xu, A. Youns and M. abua, "Autoatc Movng Object Extracton for ontent-ased Applcaton," IEEE Trans. rcuts Syste Vdeo Technology, vol. 4, no. 4, pp , 004. [3] S. Sun, D. Haynor and Y., "Seautoatc Vdeo Object Segentaton usng Vsnaes," IEEE Trans. rcut Syste Vdeo Technology, vol. 3, no., pp. 75-8, 003. [4] A. Tealp,. Tolu and E. A. Tanju, "Seautoatc Vdeo Object Segentaton n The Presence of Occluson," IEEE Trans. rcut Syste Vdeo Technology, vol. 0, no. 4, pp , 000. [5] E. Şayol, E. Güdübay and O. Ulusoy, "A Se-Autoatc Object Extracton Tool for Queryng," n Multeda Databases. In Proceedngs of the 7th Worshop on

6 Multeda Inforaton Systes (MIS '0), Vlla Orland, apr, Italy, 00. [6] T. Meer and. Ngan, "Autoatc Segentaton of Movng Objects for Vdeo Plane Generaton," IEEE Trans. rcut Syste Vdeo Technology, vol. 8, no. 5, pp , 00. [7] T. Tsag and A. Averbuch, "Autoatc Segentaton of Movng Objects n Vdeo Sequences : A Regon Labelng Approach," IEEE Trans. rcut Syste Vdeo Technology, vol., no. 7, pp , 00. [8] A. avallaro, Seantc Vdeo Object Segentaton Tracng and Descrpton, Ph.D Thess, Ecole Polytechnque Federale de Lausanne, 00. [7] D. Tollver and G. Mller, "Graph Parttonng by Spectral Roundng: Applcatons n Iage Segentaton and lusterng," n Proc. IEEE Int l onf. oputer Vson and Pattern Recognton, 006., 006. [8] M. Soelean, M. Harad and M. Purnoo, "Adaptve Threshold for acground Subtracton n Movng Object Detecton usng Fuzzy -Means lusterng," n Tencon Int'l onference, ebu, Phlppnes, 0. [9] R.. Gonzalez and R. E. Woods, Dgtal Iage Processng 3rd edton, Pearson Prentce Hall, 007. [9] H. Luo and A. Eleftherads, "Model-based Segentaton and Tracn of Head-and- Shoulder Vdeo Object for Real Te Multeda Servce," IEEE Trans. Multeda, vol. 5, no. 3, pp , 003. [0] A. Levn, D. Lschns and Y. Wess, "A losed-for Soluton to Natural Iage Mattng," IEEE Transactons on Pattern Analyss And Machne Intellgence, vol. 30, pp. -5, 008. [] T. Porter and T. Duff, "opostng dgtal ages," oputer Graphcs, vol. 8, [] A. Levn, A. Rav-Acha and D. Lschns, "Spectral attng," IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 30, 008. [3]. Lang, "Fxng Two Weanesses of the Spectral Method," n Proc. Advances n Neural Inforaton Processng Systes, 005. [4] S. Yu and J. Sh, "Multclass Spectral lusterng," n Proc. Int lonf. oputer Vson, 003. [5] L. Zeln-Manor and P. Perona, "Self-Tunng Spectral lusterng," n Proc. Advances n Neural Inforaton Processng Systes, 005. [6] A. Ng, M. Jordan and W. Y., "Spectral lusterng: Analyss and an Algorth," n Proc. Advances n Neural Inforaton Processng Systes, 00.