ISSN Vol.04,Issue.15, October-2016, Pages:

Transcription

1 ISSN Vol.04,Issue.15, October-2016, Pages: An Effcent Co-Segmentaton Algorthm for Vdeos KRISHNAIAH GOTHULA 1, P.RAJESH 2, A.RAJANI 3 1 PG Scholar, Dept of ECE(DSCE), Annamacharya Insttute of Technology, Trupath, AP, Inda. 2 Assstant Professor, Dept of ECE, Annamacharya Insttute of Technology, Trupath, AP, Inda. 3 Assstant Professor, Dept of ECE, Annamacharya Insttute of Technology, Trupath, AP, Inda. Abstract: Wth ever-ncreasng volumes of vdeo data, automatc extracton of salent object regons became even more sgnfcant for vsual analytc solutons. Ths surge has also opened up opportuntes for tang advantage of collectve cues encapsulated n multple vdeos n a cooperatve manner. However, t also brngs up major challenges, such as handlng of drastc appearance, moton pattern, and pose varatons, of foreground objects as well as ndscrmnate bacgrounds. Here, we present a cosegmentaton framewor to dscover and segment out common object regons across multple frames and multple vdeos n a jont fashon. We ncorporate three types of cues,.e., ntraframe salency, nterframe consstency, and acrossvdeo smlarty nto an energy optmzaton framewor that does not mae restrctve assumptons on foreground appearance and moton model, and does not requre objects to be vsble n all frames. We also ntroduce a spato-temporal scale-nvarant feature transform (SIFT) flow descrptor to ntegrate across-vdeo correspondence from the conventonal SIFT-flow nto nterframe moton flow from optcal flow. Ths novel spato-temporal SIFT flow generates relable estmatons of common foregrounds over the entre vdeo data set. Expermental results show that our method outperforms the state-of-the-art on a new extensve data set (VCoSeg).. Keywords: Vdeo Object Co-Segmentaton, Energy Optmzaton, Object Refnement, Spato-Temporal Scale- Invarant, Feature Transform (SIFT) Flow. I. INTRODUCTION Wth the faster growth of vdeo data, effcent and automatc extracton of the nterest object from multple vdeos s qute mportant and very challengng. Maybe these objects of nterest exhbt drastcally dfferent n ther appearance or motons. Moreover, foreground appearance or motons from varous vdeos are much dfferent, whle possbly low contrast wth the bacground. These challenges cause great dffcultes on exstng vdeo segmentaton technques [3], [11], [12], [13] whch usually beneft from vsual cues such as moton or appearance. Addtonally, these methods rely on the assumpton that the moton or appearance of object s dramatcally dstnct from bacground, whch s aganst the stuaton as we mentoned before. Moreover, the lac of tang nto account the jont nformaton between vdeos leads to unsatsfactory performance of these methods desgned for sngle vdeo on ths ssue(see Fg.1(b)). In contrast to prevous object segmentaton methods for a sngle vdeo, vdeo cosegmentaton has been proposed to extract the man common object from a set of related vdeos. Vdeo co-segmentaton utlzes vsual propertes across multple vdeos to nfer the object of nterest wth the absence of pror nformaton about vdeos or foregrounds. There are a few methods desgned for ths problem tll now [1], [2], [5]. Whle these approaches mae qute strong assumptons on the moton patterns or appearance of foreground. For example, Rubo et al. [1] mae assumptons that the foreground objects from dfferent vdeos have smlar moton patterns and smlar appearance model whch s dstnct from the bacground. Chen et al. [2] emphasze that the coherent moton of regons and smlar appearance are able to conduct the segmentaton. Addtonally, one general lmtaton of these approaches [1], [2] s that the set of vdeos s assumed to be smlar or related for foregrounds and bacgrounds. Chu and Frtz [5] treat the tas of vdeo co-segmentaton as a multclass labelng problem, but ts classfcaton results heavly rely on the chroma and moton features (see Fg. 1 (c)). Totally, these prevous vdeo co-segmentaton approaches [1], [2], [5] have two man lmtatons. Frst, both approaches abuse moton and appearance based cues and gnore the fact that there are consderable vdeos wth the common object low contrast wth the bacground. Second, n both approaches, the process of nferrng common objects does not effectvely explore the correspondence of objects from dfferent vdeos, whch s essental for the tas of vdeo cosegmentaton. These methods smply assume that the objects are smlar n moton patterns or appearance, whch s not sutable for the scene that ncludes objects wth large varatons n appearance or moton. Besdes, there are consderable vdeos that nclude some frames not contanng the common object of the whole vdeo sequence. For nstance, the foreground object moves out of camera or the swtchng between vdeo shots. However, ths general fact s gnored by most prevous wor n both vdeo object segmentaton and co-segmentaton methods. Most of methods assume that the foreground object appears n every frame, and hence they are unable to perform well for ths ssue. Ths paper presents a co-segmentaton framewor for detectng and segmentng out common objects from 2016 IJIT. All rghts reserved.

2 multple, contextually related vdeos wthout mposng above constrants. In our approach, we explore the underlyng propertes of vdeo objects n three levels: ntra-frame salency, nter-frame consstency and across-vdeo correspondence. Based on these propertes, we ntroduce a spato-temporal SIFT flow descrptor to capture the relatonshp between foreground objects. We establsh an object dscovery energy functon utlzng the spato-temporal SIFT flow and nterframe consstency to dscover the common objects. Our source code wll be publcly avalable onlne.1 Compared to exstng vdeo co-segmentaton approaches, the proposed method offers followng contrbutons: A novel vdeo co-segmentaton method s proposed for automatcally segmentng out the foreground object wth low constrant for ther appearance and moton patterns. We are the frst to fully explore the propertes of foreground object n vdeo: ntra-frame salency, nterframe consstency and across-vdeo smlarty. These mportant cues are further formulated nto our vdeo co-segmentaton framewor as the optmzaton problems. An effcent spato-temporal SIFT flow s developed to buld relable correspondences between dfferent vdeos, whch can nfer the common object over entre vdeo dataset and refne the segmentaton accuracy for objects. Appearance or motons from varous vdeos are much dfferent, whle possbly low contrast wth the bacground. Fg.1. Vdeo Co-Segmentaton. (a) Input Vdeos Where Objects Have Large Varatons. (b) Results By [2], Whch Lacs The Jont Informaton Between The Vdeos. (c) Results By Vdeo Co-Segmentaton Method of [5]. Over Fragmentaton s Vsble. Also, Parts of Foregrounds (E.G. Brd) Are Merged Into Bacground As ts Global Model Heavly Reles on The Chrome And Moton. (d) Our Vdeo Object Co-Segmentaton Results. KRISHNAIAH GOTHULA, P.RAJESH, A.RAJANI These challenges cause great dffcultes on exstng vdeo segmentaton technques [3], [1], [2], [6], [9], [10], whch usually beneft from vsual cues such as moton or appearance. Addtonally, these methods rely on the assumpton that the moton or appearance of object s dramatcally dstnct from bacground, whch s aganst the stuaton as we mentoned before. Moreover, the lac of tang nto account the jont nformaton between vdeos leads to unsatsfactory performance of these methods desgned for sngle vdeo on ths ssue (see Fg. 1 (b)). In contrast to prevous object segmentaton methods for a sngle vdeo, vdeo co-segmentaton has been proposed to extract the man common object from a set of related vdeos. Vdeo cosegmentaton utlzes vsual propertes across multple vdeos to nfer the object of nterest wth the absence of pror nformaton about vdeos or foregrounds. There are a few methods desgned for ths problem tll now [21], [22], [25]. Whle these approaches mae qute strong assumptons on the moton patterns or appearance of foreground. II. RELATED WORK We gve a short overvew of the prevous wor along two major themes: vdeo co-segmentaton and vdeo object segmentaton technques below. A. Vdeo Co-Segmentaton: Vdeo co-segmentaton has receved attentons only recently, thus there are very few methods [1], [2], [5] specally desgned for ths purpose to the best of our nowledge. Rubo et al. [1] provded an teratve optmzaton framewor to acheve such a vdeo cosegmentaton tas. Ths wor s based on a dense feature matchng process executed on regon and tube levels usng jont appearance and moton models of the foreground and bacground. Whle ths approach made qute strong assumptons that foreground objects from dfferent vdeos have smlar moton patterns and smlar appearance models whch are dstnct from bacground. Obvously, ts applcablty s lmted by ts unmatched assumptons. The wor by Chen et al. [2] utlzed the moton coherence and appearance cues to separate the common object n a par of related vdeos. However, ths method attempted to group the regons nto foreground and bacground accordng to the coherent moton and smlar appearance, whch leads to unsatsfactory performance for the vdeo wth smlar foreground and bacground motons or appearance. Moreover, both Rubo et al. [1] and Chen et al. [2] requred the nput vdeos to be smlar. Therefore, they may fal for cases that have large varatons n foreground appearance and complex bacgrounds. Chu and Frtz [5] performed multclass vdeo co-segmentaton by buldng a non-parametrc Bayesan model based on Drchlet Processes that reles on the chroma smlarty and moton dstncton constrants. As a result, the dscrmnaton power of ths model s lmted n complex scenaros. When the nput vdeos wth more common scenaro, ther results sometmes are consstent wth the regons that exhbt coherent appearance or moton nstead of a partcular object. It can be seen that vdeo object cosegmentaton s stll an emergng research problem to be ntensvely nvestgated. Internatonal Journal of Innovatve Technologes

3 An Effcent Co-Segmentaton Algorthm for Vdeos optcal flow, whch captures nter-frame moton, and conventonal SIFT flow, whch captures across-vdeos correspondence nformaton. Our algorthm has three man stages: object dscovery among multple vdeos, object refnement between vdeo pars, and object segmentaton on each vdeo sequence. 1. Object Dscovery: We use salency and spato-temporal SIFT flow to estmate common object regons n the entre vdeo dataset. In ths stage, an ntal assgnment of pxels belongs to object s performed. 2. Object Refnement: The goal s to refne the estmated object regons generated by pror step. Ths object refnement process s executed across a pars of vdeos. 3. Object Segmentaton: Snce the correct estmaton for object n each vdeo s avalable, we can model the appearance of foreground and mae segmentaton on each vdeo sequence to get more accurate results. Fg.2. Overvew of Our Object Dscovery Step. (a) Four Input Vdeos Where Brd s the Common Object. Ths Object Dscovery Process Does Not Need To Be Performed At Full Frame Rate. There Are Fve Frames Between Frame F K and Frame F K+1. (b) Salency Informaton And Spato-Temporal SIFT Flow Are Introduced Into Ths Step To Get The Common Object n Vdeo Set. (c) Output of the Object Dscovery Step s a Coarse Estmaton for the Common Object Regons n Each Frame Based on the Object Dscovery Energy Functon As n (13). B. Vdeo Object Segmentaton: The goal of vdeo object segmentaton s to detect the prmary object and extract the object from a sngle vdeo. There has been a large body of wor concentratng on ths tas last decade. Vdeo object segmentaton methods can be broadly classfed nto two categores: nteractve (supervsed) methods and automatc (unsupervsed) methods. For nteractve vdeo object segmentaton [2], [4], [5], [11] [13], [8], user nteractons and optmzaton technques employng moton and appearance constrants are often ntroduced to produce hgh qualty segmentaton results. Our method s more closed to unsupervsed vdeo object segmentaton. Unsupervsed vdeo object segmentaton ams at autonomously mergng pxels nto foreground or bacground wthn ther vdeo. Earler automatc segmentaton methods employed appearance or moton based cues for a bottom-up segmentaton. Several methods were proposed to select prmary. III. PROPOSED SYSTEM A. Overvew Our goal s to jontly segment multple vdeos contanng a common object n an unsupervsed manner. We consder ths tas as an object optmzaton process conssts of object dscovery; object refnement and object segmentaton executed on the whole set of vdeos. In ths optmzaton process, we use a spato-temporal SIFT flow that ntegrates B. Object Dscovery In ths stage, our method explores the vdeo dataset structure and assocates the global nformaton wth the ntraframe nformaton le salency to dscover the common object from multple vdeos, even n the presence of some frames wthout the common object. Three man propertes of targeted object are helpful for object dscovery: a) ntra-frame salency the pxels of foreground should be relatvely dssmlar to other pxels wthn a frame; b) nter-frame consstency the pxels of foreground should be more consstent wthn a vdeo; c) across-vdeo smlarty the pxels of foreground should be more smlar to other pxels between dfferent vdeos (wth possble changes n color, sze and poston).we propose a new spato-temporal SIFT flow algorthm that ntegrates salency, SIFT flow and optcal flow to explore the correspondences between dfferent vdeos. Thus, an object dscovery energy functon s then desgned to effectvely nfer the common objects wthout the constrants that the object must exst n each frame. An overvew of our algorthm s shown n Fg. 2. Salency of a pxel reflects how salent the pxel s, namely, the degree of ts dssmlarty wthn the mage. There are several methods n computer vson that concentrate on ths topc. We use [2] yet any other salency methods such as [3] can be ncorporated. Let V = {V1, V2, VN} be a set of N nput vdeos. Fn = {F1n, F2n... Fn,...} s a set of frames belong to vdeo Vn. We compute a normalzed object regons n object proposal doman based on the noton of what a generc object loos le. These methods beneft from the wor of object hypotheses proposals [8] [10] that offer consderable object canddates n every mage/frame. Therefore, segmentng vdeo object s transformed nto an object regon selecton problem. In ths selecton process, both moton and appearance cues are reasonably used to measure the object-ness of a proposal. In recent years, Lee et al. [6] ntroduced an alternatve clusterng process, Ma and Latec [9] attempted to model the selecton process as a constraned maxmum weght clques problem, Internatonal Journal of Innovatve Technologes

4 and Zhang et al. [2] proposed a layered drected acyclc graph based framewor. Salency map M n map for framef n. Based on ntra-frame salency property, the larger value of M n, the more lely that the pxel x = (x, y) belongs to object. Then we buld a salency term A n(x) to defne the cost of labelng pxel x for foreground l n x = 1 or bacgroundl n x = 0. A n x = exp M n x. l n x + exp 1 M n x. (1 l n x ) (1) Optcal flow [7] s represented as a 2D vector, whch reflects the moton nformaton of pxel x based on the color consstency assumpton between consecutve frames. Optcal flow algorthms can be used to estmate the nter-frame moton at each pxel n a vdeo sequence. Let V n denote the flow feld between frame F n and F n Here, a pxel x and ts moton compensated pxel x + v n(x) are smlar between two consecutve frames F n and F n + 1, whch represents the nter-frame consstency property. KRISHNAIAH GOTHULA, P.RAJESH, A.RAJANI dsregard ths Challenge and assume common object appears n every frame. Our method effectvely handles ths dffculty. One ntuton s that the frames that do not contan the common object are not consstent wth the frames that contan the object. Therefore, we further leverage the nter-frame consstency property. Based on (10), we get object-le areas and bacground areas for each frame. Suppose frame f n contans the common foreground whle f n n does not. Ther estmated object-le area should be dfferent Fg.4. Fg.3. Comparson between Our Spato-Temporal SIFT Flow and Tradtonal SIFT Flow[13]. Fg.3. shows a comparson between the proposed spatotemporal SIFT flow and tradtonal SIFT flow. Fg. 3(a) depcts two frames Fn and Fn need to be matched. Fg. 3(b) are the computed salency mas (of frame Fn ) and the salency map Mn of frame Fn. Fg.3(c) shows the result that frame Fn warped onto frame Fn accordng to tradtonal SIFT flow. The blac regon s the matched area outsde the mage range, whch s ncorrect. Fg. 3(d) gves the result of spatotemporal SIFT flow wthout salency constrant. It s vsble that spato-temporal SIFT flow s more accurate than the conventonal SIFT flow. Stll, the performance of matchng s not suffcent enough due to the dsturbance of the bacground. The correct result should be that a lon les the one n frame Fn s presented n frame F n_. Fg. 3(e) shows the result of spato-temporal SIFT flow by consderng the salency mas n Fg. 3(b), where the performance gans sgnfcant mprovement. There are many vdeos that nclude frames that do not contan the common object (e.g. the frst row of Fg. 4). Current vdeo co-segmentaton approaches Fg.4. Effectve object dscovery from multple vdeos even wth some frames not contanng the common object. The frst row shows two elated vdeo sequences and the common object plane does not appear n every frame. The object-le area of each frame estmated through (1) are resented n the second row. The bottom row shows the more correct object dscovery results through (3) wth further utlzng the nterframe consstence property. Those frames wth the rato κ 0.2 are consdered not to contan the common object, whch are mared n the red rectangles. We employ Gaussan mxture models (GMM) to characterze the common object appearance. For frame f n, the GMMs for object-le area and bacground are defned as {GMM f f n, GMM b f n }, respectvely. We ntroduce an object consstence term to measure the consstency of estmated objects n vdeo accordng to the appearance model of object. For frame f n ths object consstence term s defned as: C n x = exp P n x. l n x + exp 1 P n x. (1 l n x ) (2) Where P n (x) denotes the probablty of pxel x for foreground, whch s obtaned from GMM f f n, GMM b f n Of pror frame f n and f n. Then we add ths object consstence term nto our object dscovery energy functon. n x = 1A n x + 2M n X + 3C n x + V n (X) (3) We set parameter 1 = 2 = 3 = 50 for all the test vdeos n our experments. Snce fve or ten frames between Internatonal Journal of Innovatve Technologes

5 An Effcent Co-Segmentaton Algorthm for Vdeos frame f n the estmated GMM for frame f n s helpful for IV. SIMULATION RESULTS dentfyng whether the frame f n Contans the common object. C. Object Refnement In the prevous step, we obtan a coarse estmaton for the common object n the dataset. Based on ths, we see to obtan a more accurate estmaton for foreground object n every vdeo. Our ntuton s to remove the pxels that are smlar to bacground based on the estmaton result. Nevertheless, ths also requres determnng what foreground would loo le. To flter out bacground pxels we dvde the object-le area nto sub-regons based on ther varatons. We utlze spato-temporal SIFT flow for ths purpose. Fg. 6 llustrates the procedure of the object refnement step. Frst, a par of vdeos (Vn, Vn) s randomly selected from dataset. Ther spato-temporal SIFT flow between frames f n and f n_ s constructed. As shown n Fg. 6(c), dscontnutes of spato-temporal SIFT flow feld reflect the varaton of object structure (but not color varaton) yet robust to object detals. Ths property of spato-temporal SIFT flow feld s very mportant. Through the computaton of the dscontnutes of spato-temporal SIFT flow feld, we dvde the object-le area nto a few regons dependng on the structure varaton. Ths enables us to estmate every part of the object-le area whether belongs to foreground usng GMMs. Propertes of flow feld boundares reveal the physcal cues of object as nvestgated n the past [20], [26]. In [20], an embeddng dscontnuty detector s proposed for localzng object boundares n trajectory spectral embeddng, however ths s not sutable for our wor. In [26], an algorthm s presented to detect the moton boundary and determne whch pxels resde nsde the movng object s presented. Ths method faces dffculty when the foreground moton patterns are not dstnct. Moreover, t dvdes the frame only nto two parts, whle we want to dvde the object-le area nto multple regons based on the structure varatons. Based on the vsualzaton of spato-temporal SIFT flow feld usng [1], numerous over-segmentaton methods can be ntroduced and the object-le area can be effcently parttoned nto regons as shown n Fg. 6(d). Each pxel denotes a flow vector where the orentaton and magntude are represented by the hue and saturaton of the pxel, respectvely. D.Object Segmentaton by Optmzaton Once the correct estmatons for foreground of each vdeo are obtaned, a graph-cut based method s employed to get per-pxel segmentaton results. Recall our defnton of = {f 1 n, f 2 n,.., f n,. } s that we select frame f n every other fve or ten frames from VdeoV n. After the object refnement process we get more correct estmaton for common object and update the appearance model of the object and bacground GMM f f K n, GMM b K f n for frame f n,whch can be used to conduct the segmentaton n next fve or ten frames of f n. For framef n, we obtan the lelhood of pxel x for foreground P n (x) usng our appearance models estmated by ts temporally nearest frame off n. Fg.5. Input Images. Fg.6. Output Images. V. CONCLUSION We presented a robust vdeo co-segmentaton method that dscovers the common object over an entre vdeo dataset and segments out the objects from the complex bacgrounds. Salency, moton cues and SIFT flow are ntegrated nto our spato-temporal SIFT flow to explore the relatonshps between foreground objects. Furthermore, we formulate the vdeo co-segmentaton problem as an object optmzaton process, whch progressvely refne the estmaton for object n three steps: object dscovery, object refnement and object segmentaton. Both the quanttatve and qualtatve expermental results have shown that the proposed algorthm creates more relable and accurate vdeo co-segmentaton performance than the state-of-the-art algorthms. Unle prevous wor, we emphasze that object dscovery process should be robust to foreground varatons n appearance or moton patterns, whch extends the applcablty of our cosegmentaton method. VI. REFERENCES [1]S. Baer, D. Scharsten, J. P. Lews, S. Roth, M. J. Blac, and R. Szels, A database and evaluaton methodology for optcal flow, Int. J. Comput. Vs., vol. 92, no. 1, pp. 1 31, Mar Internatonal Journal of Innovatve Technologes

6 KRISHNAIAH GOTHULA, P.RAJESH, A.RAJANI [2]X. Ba, J. Wang, D. Smons, and G. Sapro, Vdeo Snap Cut: Robust vdeo object cutout usng localzed classfers, ACM Trans. Graph., vol. 28, no. 3, Aug. 2009, Art. ID 70. [3]L. S. Slva and J. Scharcans, Vdeo segmentaton based on moton coherence of partcles n a vdeo sequence, IEEE Trans. Image Process., vol. 19, no. 4, pp , Apr [4]Y. Huang, Q. Lu, and D. Metaxas, Vdeo object segmentaton by hyper graph cut, n Proc. IEEE CVPR, Jun. 2009, pp [5]J. Yuen, B. Russell, C. Lu, and A. Torralba, Label Me vdeo: Buldng a vdeo database wth human annotatons, n Proc. 12th IEEE ICCV, Sep./Oct. 2009, pp [6]W. Brendel and S. Todorovc, Vdeo object segmentaton by tracng regons, n Proc. 12th IEEE ICCV, Sep./Oct. 2009, pp [7]T. Brox and J. Mal, Large dsplacement optcal flow: Descrptor matchng n varatonal moton estmaton, IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 3, pp , Mar [8]I. Endres and D. Hoem, Category ndependent object proposals, n Proc. ECCV, 2010, pp [9]B. Alexe, T. Deselaers, and V. Ferrar, What s an object? n Proc. IEEE CVPR, Jun. 2010, pp [10]J. Carrera and C. Smn chsescu, Constraned parametrc mn-cuts for automatc object segmentaton, n Proc. IEEE Conf. CVPR, Jun. 2010, pp [11]D. Tsa, M. Flagg, and J. Rehg, Moton coherent tracng wth multlabel MRF optmzaton, n Proc. BMVC, 2010, pp [12]T. Wang and J. Collomosse, Probablstc moton dffuson of labellng prors for coherent vdeo segmentaton, IEEE Trans. Multmeda, vol. 14, no. 2, pp , Apr [13]M. Grundmann, V. Kwatra, M. Han, and I. Essa, Effcent herarchcal graph-based vdeo segmentaton, n Proc. IEEE Conf. CVPR, Jun. 2010, pp Internatonal Journal of Innovatve Technologes