Unsupervised Segmentation of Stereoscopic Video Objects: Proposal. and Comparison of Two Depth-Based Approaches

Transcription

1 Unsupervise Segmentation of Stereosopi Vieo Objets: Proposal an Comparison of Two Depth-Base Approahes Klimis S. Ntalianis an Athanasios S.Drigas Net Meia Lab, NCSR Demokritos, Athens, Greee Abstrat In this paper two effiient unsupervise vieo objet segmentation approahes are propose an thoroughly ompare in terms of omputational ost an quality of segmentation results. Both methos are base on the exploitation of epth information, estimate for stereosopi pairs of frames. In partiular, in both shemes an olusion ompensate isparity fiel is initially ompute an a epth map is generate. Then a epth segments map is proue by inorporating a moifie version of the multiresolution Reursive Shortest Spanning Tree segmentation algorithm (M-RSST). Next onsiering the first Constraine Fusion of Color Segments (CFCS) approah, a olor segments map is reate, by applying the onventional M- RSST to one of the stereosopi hannels. In this ase vieo objets are extrate by fusing olor segments aoring to epth similarity riteria. In the seon metho an ative ontour is automatially initialize onto the bounary of eah epth segment, aoring to a fitness funtion that onsiers ifferent olor areas an preserves the shapes of epth segments' bounaries. Then eah point of the ative ontour is assoiate to an attrative ege point an a greey approah is inorporate so that the ative ontour onverges to its final position to extrat the vieo objet. Several experiments on real life stereosopi sequenes iniate the promising performane of the propose methos as effiient unsupervise vieo objet segmentation tools. Extensive omparisons are also performe onerning spee an auray issues of the propose shemes. Keywors: unsupervise vieo objet segmentation, isparity fiel, epth map, M-RSST, ative ontour, attrative ege point, greey approah, performane evaluation.

2 1. Introution Although blok-base vieo oing stanars suh as MPEG-1/2 an H.261/3 have serve as orner stones to many onventional appliations suh as vieo ompression/transmission, they meet several limitations when averting to urrent nees. During the last eae bursting inrease in popularity of multimeia appliations base on ontent interation, has le to the emergene of new stanars as MPEG- 4/7. The MPEG-4 stanar introue the onepts of vieo objets (VOs) an vieo objet planes (VOPs). A VO orrespons to a semantially meaningful objet suh as a human, a house et., while a VOP is an arbitrarily shape region originating from the projetion of a VO onto an image plane. These ieas woul benefit the implementation of systems for ontent-base inexing, iretive vieo browsing/summarization, objet-base oing, transmission an rate ontrol, or multimeia synthesis an manipulation. Aitionally, although 2-D sequenes urrently prevail vieo arhives, the eman for stereosopi sequenes is lately inreasing sine they enhane ommuniations, offering spetaular shows that highly approah real experienes. This is ue to the fat that they provie the sensation of epth, whih is an inherent harateristi of stereosopi sequenes an its exploitation an lea to the implementation of more effetive ontent segmentation algorithms. However the main isavantage of stereosopi vieo is its inrease banwith an apaity emans for transmission an storage. For these reasons in this paper a 3- D to 2-D sheme is propose, where the sequene is apture by a stereosopi amera, unergoes high-level analysis towars extration of VOPs an finally an MPEG-4 ompatible 2-D sequene is proue. On the other han, the visual part stanarization phase of MPEG-4 was oriente on eveloping unsupervise VOP segmentation algorithms [1], but i not result into a globally appliable sheme. This is ue to the fat that semanti objets usually onsist of multiple regions with ifferent olor, texture an motion an although humans an onnet the suitable regions almost effortlessly, a mahine annot effetively perform the same task, whih still remains a hallenging problem. Exeptions an be generally lassifie into two main ategories: (A) The first ategory inlues ases where onitions are known suh as graphis, where ontent esription is a-priori available, or the ase of vieo sequenes proue using the hroma-key tehnology. (B) The seon ategory inlues ases where VOP segmentation shoul not or annot be performe, as in ases of frames with a large number of similar vieo objets e.g. a row, or when the overing region of a vieo objet oupies most of the frame area. In 2D-vieo, low-level features, suh as olor, texture an motion information are use as ontent esriptors [2]. In stereosopi vieo however, the problem of ontent-base segmentation an be aresse more effetively as epth information an be reliably estimate an sine eah vieo objet usually oupies a istint epth plane [3]. Several mainly supervise tehniques have been propose in literature for VOP segmentation. Some harateristi examples inlue: (A) the approah in [4] where multiimensional analysis of several image features is performe by a spatially onstraine fuzzy C-means algorithm. (B) The work esribe in [5], where initially the VOP is etete using a ombination of human assistane an a morphologial watershebase segmentation tehnique. Experimental results are obtaine using a low-en Pentium 133-MHz mahine. The average time for traking is 2 s ( Mobi ), 3 s ( Akiyo ), an 5 s ( Foreman ). (C) The works in [6], [7], where an ative ontour is manually initialize aroun the objet of interest an moves towars

3 the iretion of energy minimization. In ase of [6] the 2-D GVF omputations were implemente using MATLAB oe. For an pixel image on an SGI Inigo-2, typial omputation time is 420 s for the GVF fores, whih an be reue to 53 s, when written in C. On the other han in ase of [7], in all emonstrate ases the eformation proess took a fration of a seon on a SUN-IPX workstation, for less than 100 verties. However, in the aforementione supervise methos the time of user interation (to group regions or properly initialize an ative ontour) is generally not estimate, an an be from several seons to minutes. For these reasons several vieo appliations, annot inlue supervise tehniques. On the other han some unsupervise shemes have been presente, base on information fusion tehniques. In [8] a multisale graient tehnique followe by the watershe segmentation algorithm is use for olor segmentation, while motion parameters of eah region are estimate an regions with oherent motion are merge to form the objet. Experimental results show that segmentation an traking of 12 frames ( pixels) takes about 30 s of CPU time on a Sun workstation. Aitionally, as state in the onlusions, an objet an be split into several objets (oversegmentation) aoring to motion ompensation error. In [9] a binary moel of the objet of interest is erive, the points of whih onsist of ege pixels etete by the Canny operator. The moel is initialize an upate aoring to motion information, erive by inorporation of Change Detetion Masks (CDMs) or morphologial filtering. Another sheme is presente in [10] where temporal an spatial segmentation are performe. Temporal segmentation is base on intensity ifferenes between suessive frames inluing a statistial hypothesis test, while spatial segmentation is arrie out by the watershe algorithm. Information about omputational omplexity is not provie, while auray is heavily influene by shaow effets, refletions, an noise. Furthermore if there is insuffiient motion, parts of objets might be missing in the orresponing VOPs. The work presente in [11] is base on loalizing moving VOPs over the MPEG ompresse omain. Initially maroblok MVs are utilize to ientify the loations of moving VOPs an then DC oeffiients are onsiere so as to ahieve finer bounary segmentation. All aforementione tehniques are base on motion information an an proue satisfatory results mainly in ases where bakgroun an foregroun have ifferent motion parameters an olor segments belonging to the foregroun VOP have oherent motion. However in real life sequenes there are ases where motion information is not aequate for VOP segmentation. These ases inlue: (A) Insuffiient motion between suessive frames (slowly progressing senes). (B) Senes onsisting of more than two objets (foregroun/bakgroun) having ifferent motion parameters. (C) Cases where olor segments of the region of interest o not have oherent motion (non-rigi objet) (D) Senes where the bakgroun an the vieo objet of interest have similar motion. In this paper, two effiient unsupervise vieo objet segmentation tehniques are propose, whih take as input stereosopi vieo sequenes an provie at the output MPEG-4 ompatible 2-D sequenes. Both approahes share in ommon the exploitation of epth information, as in many ases most points of a vieo objet have the same epth. Aoring to this onsieration, initially a vieo sequene is analyze an the isparity fiel, olue areas an a epth map are estimate. In a seon step, a segmentation algorithm is inorporate to partition the olusion ompensate epth map into homogeneous regions. In this paper a moifie version of the multiresolution Reursive Shortest Spanning Tree (M-RSST) segmentation

4 algorithm is aopte. This sheme, apart from aelerating the segmentation proess ompare to the onventional M-RSST it also prevents oversegmentation, whih is not esirable in ases where effiient visual ontent esription shoul be aomplishe. Then, for the first Constraine Fusion of Color Segments (CFCS) sheme, one of the image hannels (left or right) is partitione into olor segments by the onventional M-RSST. The CFCS sheme is base on the iea that olor segments provie aurate objet ontours, while eah epth segment roughly approximates a vieo objet. In the final step of the CFCS metho, the olor map is projete onto the epth map an olor segments with similar epth are fuse, so that vieo objets with reliable bounaries are extrate. The seon metho is base on ative ontours, whih are automatially initialize on the bounary of eah epth segment, in ontrast to most existing methos where initialization is manually performe. Firstly the initial points of the ative ontour are unsupervisely selete by means of a hybri polygonal shape approximation tehnique, whih also takes into onsieration regional olor variations. Seonly eah ative ontour point is assoiate to an attrative ege point that exerises an attration fore to the ative ontour point. Finally a greey algorithm is inorporate an the ative ontour evolves from its initial position to the final minimal energy position to extrat the vieo objet. The presente experimental results on real life stereosopi vieo sequenes iniate the prominent performane of both propose shemes as ontent segmentation tools. Furthermore omprehensive omparison tables exhibit the avantages an isavantages of eah sheme regaring omputational ost an vieo objet segmentation auray. This paper is organize as follows: In Setion 2 epth map estimation an segmentation are ompatly presente. Desription of the CFCS vieo objet segmentation metho together with implementation issues are isusse in Setion 3 while the ative ontour-base sheme is investigate in Setion 4. Experimental results are presente an analyze in Setion 5 together with a etaile isussion of the avantages an isavantages of eah metho. Finally, setion 6 onlues this paper. 2. Unsupervise Estimation of the Depth Segments Map A blok iagram esribing the moules for prouing the propose epth segments map is illustrate in Figure 1. As it an be observe, the proess onsists of three main moules: isparity fiel estimation, olusion etetion/ompensation an epth fiel segmentation. 2.1 Disparity Fiel Estimation Let us assume that a point w with worl oorinates (X, Y, Z) is perspetively projete onto the two image planes of a stereosopi apturing system, proviing point (x l, y l ) onto the left plane I l an point (x r, y r ) onto the right plane I r. Then the following relations between the two points (x l, y l ), (x r, y r ) an epth Z an be obtaine [12]: ( s x Z b y Z x l ) y l r, r, ( xl s) Z bs ( xl s) Z bs (1) with s = sin, = os, s = sin( /2), = os( /2) where λ is the foal length an θ is the angle forme by the onverging optial axes of the two ameras (fixe amera parameters).

5 Equation (1) iniates that, if the orresponene between pixel (x l, y l ) belonging to the left hannel an pixel (x r, y r ) belonging to the right hannel is known, then epth Z of point w an be straightforwarly estimate. In this iretion let us efine as x( xl, yl ) xr xl ( y l l r l ( x, y ) y y ) the horizontal (vertial) left to right isparity for a given point (x l, y l ) on image plane I l, where (x r, y r ) is the orresponing point of (x l, y l ) on image plane I r. Using equation (1) the horizontal an vertial isparity fiels, x an expresse as: y an be [ ( s x x x s Z b x s x y x x l ) l ( l )] ( l ) x x( l, l ) r l ( xl s) Z bs (2a) [ ( x s y Z bs y x y y y l )] l l y y ( l, l ) r l ( xl s) Z bs (2b) If isparity is known, equations (2a) an (2b) reue to an overetermine linear system of two equations with a single unknown (Z) an a least-squares metho an be applie for estimation of epth Z [13]. In this paper isparity fiel estimation is performe by minimizing a ost funtion onsisting of a isplae frame ifferene (DFD) relate term, an a spatial onsisteny riterion for smoothing the isparity fiel as propose in [14]. 2.2 Olusion Detetion/Compensation an Depth Fiel Segmentation In the previous analysis it was assume that every point (x l,y l ) of I l has a orresponing point (x r,y r ) on I r. However, ue to the ifferent viewpoints of the two ameras, there may be areas of I l that are olue in I r. Olusion etetion is aomplishe by loating regions of I l where horizontal isparity ontinuously ereases with a slope approximately equal to 1 [15]. Then, olue areas are ompensate by keeping isparity fiel onstant an equal to the maximum isparity value of the olue region [14]. One an olusion ompensate epth fiel is estimate the epth segmentation moule of Figure 1 is ativate. In this paper a moifie version of the multiresolution Reursive Shortest Spanning Tree segmentation algorithm (M-RSST) [16], is inorporate for epth segmentation. The M-RSST reursively applies the RSST to images of inreasing resolution an the RSST iteration phase terminates when either the total number of segments or the minimum link weight reahes a target threshol. The minimum link weight is preferable sine it results in ifferent number of segments aoring to the image ontent. In the propose moifie sheme, the RSST phase of the M-RSST algorithm is applie only to the lowest resolution level of the epth map, an the results are just propagate to the highest level without any ajustment, sine exat bounaries of vieo objets annot be aurately etete even if higher levels are proesse. 3. The Constraine Fusion of Color Segments (CFCS) approah The Constraine Fusion of Color Segments (CFCS) approah utilizes olor an epth information in orer to perform vieo objet extration. The first step of the propose sheme inlues partitioning of one of the image hannels (left or right) into olor segments. Now the onventional M-RSST algorithm is inorporate, where initial segmentation at the lowest resolution level is not just propagate but it is also ajuste at higher levels [16]. After prouing a olor segments map, olor segments an be fuse aoring to epth similarity riteria to provie the vieo objets. The iea of information fusion is justifie in our

6 ase, sine usually vieo objets are ompose of regions loate at the same epth plane. However, objet bounaries (ontours) annot be ientifie with high auray by a epth segmentation algorithm, mainly ue to erroneous estimation of the isparity fiel, even after it has been improve by the olusion etetion an ompensation algorithm. On the ontrary, segmentation base on olor homogeneity riteria provies reliable objet bounaries, but in most ases oversegments a vieo objet into multiple regions. For this reason, VOP segmentation is aomplishe by merging together olor segments aoring to epth similarity riteria as esribe next. Let us assume that K olor segments an K epth segments, enote as S i, i 1,2,, K an i 1,2,, K respetively, have been extrate by applying the M-RSST an the moifie M-RSST segmentation algorithms on the olor an epth intensity planes respetively. Segments S i an S i, S i satisfy the mutual exlusiveness onition, i.e., i k S S for any i, k 1,2,, K, i k an, similarly, i k S S for any i, k 1,2,, K, i k. Let us also enote by G an segmentation masks respetively, whih are given as the union of all olor (epth) segments as G K i 1 i K i 1 i G the olor an epth S, G S (3) Assuming two ifferent segmentation masks over the same lattie, say be efine, whih projets a segment of the first segmentation mask, i.e., G an G, an operator p ( ) an S i, on the seon mask, i.e., G, p( S, ) i G Sk suh that S k arg max a( S j j 1,..., K S i ), (4) where a ( ) is the area, i.e., the number of pixels, of a segment. Equation (4) iniates that the projetion of a i olor segment, say the i-th ( S ), onto the epth segmentation mask, G, returns that epth segment, S k, among all K available, with whih the respetive olor segment S i has the maximum area of intersetion. As a result, the projetion operator atually maps a olor segment to a epth segment, i.e., Using the projetion operator, eah olor segment of Si S k. G is assoiate to one epth segment an all olor segments belonging to the same epth segment ompose a group. More speifially, K lasses are reate, say C i, i 1,2,, K, eah of whih orrespons to a epth segment S i an ontain all olor segments that are projete on S i, i.e., C, i { S j : p( S j, G ) Si }, i 1,2, K (5) Using lasses whih belong to the same lass segments, say S i, C i, the final vieo objet segmentation mask is obtaine by merging all olor segments, C i. Let us enote as G, this final segmentation mask onsisting of K=K i 1,2,, K. These segments are generate as: S S i g, i 1,2,, K suh that S g Ci (6)

7 while the final mask G, is given as the union over all segments S i, i 1,2,, K, that is G K S i (7) i 1 Equation (7) enotes that olor segments are merge into K=K new segments aoring to epth similarity. A graphial example of the propose CFCS approah is presente in Figure 2, whih esribes the ase of a shuttle. In Figure 2(a) the olor segments mask an be viewe while in Figure 2(b) the respetive epth segments map is presente. In Figure 2() olor segments are projete onto the epth map an eah olor segment is assoiate to one epth segment aoring to the projetion operator p ( ). The final vieo objets segmentation mask an be observe in Figure 2(). 4. The ative ontour sheme Two-imensional eformable moels, also known as ative ontours or snakes, were originally propose by Kass et. al. [17]. Τhe energy funtional of an ative ontour an be forme as: 1 1 * Ea E ( v( s)) [ E ( v( s)) E ( v( s)) E ( v( s))] s (8) 0 a 0 int image on where E int represents the internal eformation energy of the ative ontour ue to bening or isontinuities, E image expresses the image fores an E on allows for external onstraint fores. In the propose sheme the energy to be minimize an be expresse as: * Ea w1 Eint w2e ege (9) where E int is the internal energy an E ege is a kin of onstraints energy originating from the ege map of eah epth area. Furthermore w 1 an w 2 weight the two energy fators of equation (9) an ifferent weights' ajustment an proue a wie range of ative ontour behaviors. 4.1 Calulation of Energy Fators The internal spline energy an be written as: E int =(α(s) v s (s) 2 + β(s) v ss (s) 2 )/2 (10) The above equation ontains a first-orer term that will have larger values at urves gaps an a seon-orer ontinuity term that will be larger at urves benings. The values of α an β ontrol the strething an bening potentials at eah point of the ative ontour. In this paper the energy ue to strething (v s (s)) for two neighboring ative ontour points P i-1 =(x i-1,y i-1 ) an P i =(x i,y i ) is ompute using the onventional formula: v s (s) 2 = P i /s 2 P i -P i-1 2 =(x i -x i-1 ) 2 +(y i -y i-1 ) 2 (11) On the other han a new metho for estimating bening energy (term v ss (s) of Equation (10)) is propose, whih simplifies the hanling an usage of the bening term. More speifially let us suppose that the bening energy of a ranom ative ontour point P i shoul be alulate. For this reason the previously proesse point P i-1 an the next point to be proesse P i+1 are also onsiere (Figure 3). Using these three points two lines are forme, line ε 1 efine by points P i-1 an P i an line ε 2 efine by points P i an P i+1. Then the angle ω between ε 1 an ε 2 an be alulate by:

8 2 1 artan (12) where ω (0 π) an λ 1, λ 2 are the iretion oeffiients of lines ε 1 an ε 2 respetively. As illustrate in Figure 3 two ases exist, in ase 1 the estimate angle ω is insie the triangle ^ P P P ( in -angle) while in ase 2 i 1 i i 1 the estimate angle ω is outsie ( out -angle). Thus in ase 1 the angle θ to be use is the ompute angle (θ=ω) while in ase 2, the angle θ is given by: θ= (13) -ω Then the bening energy is estimate using the following equation: v ss (s) 2 = 2 1 R (14) where R is a normalization parameter that maps angle θ to the interval [0 255]. On the other han, onsiering energy fator E ege, the iea of attrative fiels/fores for ative ontour moels has been presente in literature [6], [18]. Attrative fiels/fores usually exerise fores onto ative ontours so as to evolve faster to the minimum energy positions, while avoiing getting trappe to loal minima. However the generation of a fiel is highly time onsuming an thus it is iffiult to be inlue into fast vieo objet segmentation shemes. For example, it takes approximately 53 seons for the GVF fiel to be generate, onsiering an image of size N= pixels, for oe written in C an assuming about 250 iterations (estimation for SGI Inigo-2 mahine) [6]. For this reason an onsiering the speial onitions of our problem (that epth segment ontour is near to vieo objet ontour), in our sheme an attrative ege points iea is propose, whih an be implemente muh faster than estimating an attration fiel an an provie satisfatory results. An attrative ege is a pixel of the ege map estimate insie the visual ontent efine by a epth segment, whih is assoiate to an ative ontour point an attrats it. Then the onstraints energy E ege is estimate aoring to the istane of eah ative ontour point from its attrative ege. Both ege map generation an etetion of attrative ege points are esribe next. Let s i be the i-th epth segment of a epth segments map, the area of whih efines a speifi visual ontent Vs i onto the respetive image hannel it has been erive from. Then ege etetion is performe insie Vs i an the estimate ege map is filtere so that isolate eges are isare, prouing a filtere ege map e init_f. This ege map is use for seletion of the attrative ege points aoring to the following iterative algorithm: Step 1: Seletion of an ative ontour point P i. Step 2: Detetion of the speifi ege point of e init_f that has minimum istane from point P i. Step 3: If minimum istane is greater than a threshol thr then proee with the next ative ontour point an repeat proess from Step 1. Step 4: Else, mask aaptation (3 3) entere at the minimum istane ege point. Step 5: Computation of the number of ege points insie the mask (NF).

9 Step 6: If NF < T num, where T num ontrols the ensity of eges insie eah region, then etetion of the seon minimum istane ege point an iteration from Step 3, else urrent minimum istane point beomes attrative ege for the proesse ative ontour point. If none of the ege points or only ege points loate farther than the istane threshol thr satisfy the NF T num onition then the ative ontour point is not assoiate to an attrative ege an moves only aoring to E int uring onvergene. The proess is terminate after all ative ontour points are onsiere. Finally the energy E ege for an ative ontour point (x i, y i ) is estimate by: E ege (x i, y i ) = B* i (x i, y i ) (15) where i (x i, y i ) is the Euliean istane of the ative ontour point from its attrative ege an B is a onstant that maps i in the interval [0 255]. 4.2 Unsupervise Ative Contour Initialization In the propose sheme an ative ontour is unsupervisely initialize onto the bounary of eah epth segment, in ontrast to most existing shemes where ative ontour initialization is manually performe. The metho an be ivie into three main moules: (a) Detetion of the bounary points for eah epth segment, (b) Determination of a speifi orering for the etete points an () Seletion of the initial ative ontour points from the set of orere bounary points. Bounary points of a epth segment are those points loate at the borer of the epth segment an an be iretly foun. However, as the set of these points oes not satisfy any orering, they annot be straightforwarly use for ative ontour initialization, sine energy alulation requires the formation of orering for the points omprising the ative ontour. For this reason orering is establishe by fining the neighbors of eah bounary point, so as to effiiently esribe the shape of the epth segment uner onsieration. Towars this iretion let us enote by P i =(x i,y i ) an initially selete ranom bounary point name urrent point. Then starting from this point an moving in a lokwise manner, an orer of the bounary points is reate. In partiular the proess is iterative an for every iteration the following ations are performe: i) Mask aaptation of size 3 3, entere at urrent point. ii) Detetion of those points insie the mask that belong to the epth bounary. iii) Seletion of the nearest neighbor point, whih beomes urrent point. The proess is terminate when starting from an initial point (x i,y i ) it results to the same point (lose loop). Finally the set of orere bounary epth segment points is enote as SD= {P 1, P 2,,, P N }, where P 1 is the initial point an P N the final point (P N+1 P 1 ) Initial Points Seletion After fining set SD, links are generate between the points of this set aoring to the estimate orer, i.e. point P 1 is linke to point P 2, point P 2 is linke to point P 3, an point P N is linke to point P 1, resulting in the P 1 P 2 P N P 1 polygon. However, this polygonal line annot be use as initial ative ontour sine a large omputational ost woul be inue uring onvergene (ue to the large number of noes). For this reason an sine in the speifi appliation there is high reunany in the set of the epth bounary points,

10 only a small part of these points shoul be selete to form the initial ative ontour. Towars this iretion linear pieewise approximation tehniques an be aopte to proue a polygon resembling the original epth bounary, while rastially reuing the number of epth bounary points. More partiularly the target of the propose sheme is to selet the minimum number of points that best esribe the shape of the epth ontour, aoring to a threshol. The threshol affets onvergene time sine the number of ative ontour points is strongly relate to this threshol. However as the final polygon (initial ative ontour) epens on the orer that points are isare an in orer to keep points belonging to ifferent areas, the mean olor ifferene MC between triplets of ajaent points guies the proess. Consiering that P i-1, P i an P i+1 are three neighboring points an the image is in RGB format, the mean olor ifferene MC i for point P i is efine as: R P) R( P ) R( P) R( P ) G( P) G( P ) G( P) G( P ) B( P) B( P ) B( P) B( P ) ( 1 i i 1 i i 1 i i 1 i i 1 i i 1 i i MC i (16) 2 In the propose sheme an iterative merging tehnique is inorporate, where starting from polygon P 1 P 2 P N P 1 an by isaring points aoring to olor homogeneity an shape istane riteria, the initial ative ontour is proue. More speifially let SD= {P 1, P 2,, P N } enote the orere set of points omprising the noes of the onstrute polygon, like the one epite in Figure 4. In orer to effetively isar noes of the polygon it is essential to efine error riteria that measure the quality of fitness of the polygon. In this iretion an without loss of generality, if point P 2 is isare then istane f 2 = P 2 -K 2 an be use as the approximation error of the urve. In the ase uner stuy a maximum approximation error f max is aopte as fitness riterion. Then the iterative proess esribe below is ativate: Step 1. For eah point P i of set SD, fin MC i (Eq. 16) an alulate the mean istane D i from its two ajaent points P i-1, P i+1. Step 2. Selet point of set SD, say P i, with the minimum MC i value an alulate f i. Step 3. If f i > f max, or D i > D max, where D max is a onstant regulating the gaps between ajaent points, selet the point of set SD that possesses the next minimum MC value an repeat Step 3. Step 4. If f i f max AND D i < D max, isar P i, realulate the MC an D values for the rest of the points in the neighborhoo of P i, upate set SD an repeat proess from Step 2. Step 5. If no point satisfies the f i f max AND D i < D max onition then terminate proess. After termination of the proess the remaining points onstitute the initial points of the ative ontour. In our experiments values of f max an D max are properly selete so that a small number of points is kept (about 2-2.5% of the points of initial set SD) to reue the omputational ost, while proviing a well-positione initial ative ontour for vieo objet segmentation. 4.3 Greey Algorithm: A Fast Approah Towars Ative Contour Convergene After initialization, onvergene of the ative ontour to the minimum energy position is investigate. Many solutions to this problem have been propose in literature. The solution propose in [17] an be erive by inorporating variational alulus tehniques. However it has been shown in [19] that these tehniques present several problems suh as: (A) the solution an reveal numerial instabilities an (B)

11 points show a teneny to pile up on strong portions at an ege ontour. Amini et al. [19] propose an algorithm for the ative ontour moel using ynami programming, an approah that is more stable an allows the inlusion of har onstraints in aition to the soft onstraints inherent in the formulation of the funtional. This metho though is slow having omplexity O(nm 3 ), where n is the number of points of the ative ontour while m is the size of the region, insie whih a point an move uring a single iteration. In this paper a greey algorithm approah is aopte [20], whih allows the ative ontour to quikly onverge to its final position. Performane of this approah is omparable to ynami programming, retains stability improvements an flexibility an allows the inlusion of har onstraints, while simultaneously reues exeution time to an orer of O(nm). The quantity being minimize uring the greey algorithm phase is given by Equation (9). In orer to minimize this overall energy, eah point of the ative ontour starts moving on a gri. The gri is entere at the proesse point an overs the 3 3 neighborhoo. The energy funtional is ompute for every point in this area an the ative ontour point moves to the position of the gri that: (a) has the least energy ompare to the rest seven pixels of the gri an (b) has less energy than the urrent position of the ative ontour point. If these onitions are not satisfie then the ative ontour point stays at its urrent position an the algorithm ontinues with the next point. The algorithm terminates if no position hanges our uring an iteration. 5. Experimental Results In this setion, performane of the propose shemes is investigate while a etaile omparison of the vieo objet segmentation results is also provie. Results have been obtaine using stereosopi pairs taken from the 3-D stereosopi television program Eye to Eye [21], of total uration 25 minutes (12,739 frames at 10 frames/se). The sequene was proue in the framework of the ACTS MIRAGE projet [22] in ollaboration with AEA Tehnology an ITC. Stuio shots were exeute using Europe s stereosopi stuio unit, whih was evelope jointly by AEA Tehnology an Thomson Multimeia within the earlier RACE DISTIMA projet [23], while loation ation shots were apture using a speial lightweight an rugge stereo amera built for the ITC by AEA Tehnology. 5.1 Depth Segments Map Generation Initially a epth segments map is proue for eah examine stereo pair. After examining several stereo pairs an for presentation reasons, three harateristi stereo pairs are selete, to illustrate the performane of the propose algorithms. In partiular the left hannels are epite in Figures 5(a), () an (e) while the respetive right hannels are shown in Figures 5(b), () an (f). In the first step an olusion ompensate isparity fiel is estimate for eah stereo pair an a epth fiel is straightforwarly generate as esribe in setion 2. The olusion ompensate isparity fiel an the orresponing epth map of the first stereo pair are shown in Figures 6(a) an (b) respetively. Similar results for the other two stereo pairs are illustrate in Figures 6()-() an 6(e)-(f). Next the moifie M-RSST segmentation algorithm is applie to eah epth fiel to provie a epth segments map. More partiularly multiresolution eomposition is performe, so that a hierarhy of epth

12 fiels I(0), I(1),, I(L 0 ) is onstrute. I(1) is of size M 0 /2 x N 0 /2, while I(L 0 ) of size M 0 /2 L 0 x N 0 /2 L 0, where M 0 N 0 is the size of the full resolution epth fiel. In our experiments the lowest resolution level has been selete to be L 0 =3 (i.e., blok resolution of 8 8 pixels). Afterwars, aoring to the moifie M-RSST the RSST phase of the algorithm is applie only to the lowest resolution level an the results are just propagate to the highest level, without performing any ajustment at the intermeiate levels. This moifiation le to a more than 5 times aeleration in the performe experiments, ompare to the onventional M-RSST algorithm an more than 300 times ompare to the RSST, proviing at the same time a goo quality of epth segmentation results. Depth segmentation results for epth fiels of Figures 6(b), () an (f) are presente in Figures 7(a), 7() an 7(e) respetively. As it an be observe, bounaries present bloky artifats, something that was expete. Here it shoul be mentione that bounaries of vieo objets annot be aurately etete even if higher levels are proesse, whih justifies the omplexity-reuing moifiation of the onventional M-RSST algorithm. To better eluiate this iea, results provie by the onventional M-RSST algorithm are also epite in Figures 7(b), 7() an 7(f). Furthermore, from the results of Figure 7 it beomes lear that eah epth segment roughly approximates a vieo objet. 5.2 Segmentation Results of the CFCS Approah Consiering the CFCS approah olor segments are fuse base on epth homogeneity riteria. For this reason, after prouing a epth segments map, a olor segments mask is also proue for eah stereo pair. Towars this iretion the onventional M-RSST segmentation algorithm is inorporate, where results of the lowest resolution level are not just propagate but also ajuste at higher levels [16]. This is essential as olor segments aurately etet the bounaries of vieo objets. The multiresolution olor segmentation proess is illustrate in Figure 8 for frame of Figure 5(a). More speifially again an initial lowest resolution level of L 0 =3 has been aopte. In this figure for better apprehension an evaluation reasons, eah olor segment is an area surroune by a white ontour an not a fragment possessing the average olor intensity of the area. In partiular, Figure 8(a) presents the results of the lowest resolution level (L 0 =3), where blok resolution aroun the objet bounaries is evient. Then, initial segmentation is propagate to the following resolution level by splitting eah bounary pixel (or 8 8 blok) into four new segments (of size 4 4) aoring to the M-RSST algorithm. This is presente in Figure 8(b). As shown, the total number of segments for the next iteration of M-RSST is onsierably reue ompare to the initial number of segments of the onventional RSST at the same resolution level. Thus, a signifiant reution of the omputational omplexity is ahieve. Color segmentation at resolution level 2 (L 0 =2) is epite in Figure 8(), where, as observe, objet bounaries have been refine. Similarly, Figure 8() illustrates the bounary pixels (or 4 4 bloks) that are split into four new segments of size 2 2, while Figure 8(e) shows the final olor segments mask. Similarly olor segmentation results for Figures 5() an (e) are presente in Figures 10(a) an 11(a) respetively. As observe in all ases, olor segments aurately etet the bounaries of vieo objets but oversegment them into multiple regions. On the other han eah epth segment roughly approximates a vieo objet proviing an inorret ontour. For this reason olor segments are projete onto the epth segments map an they are fuse aoring to epth similarity riteria as esribe in setion 3. More

13 speifially in Figure 9(a) projetion of the olor segments mask of Figure 8(e) onto the epth map of Figure 7(a) an be observe. White lines orrespon to the ontours of olor segments, while eah region with ifferent olor represents a ifferent epth segment. Then in Figure 9(b) fusion of olor segments belonging to the same epth segment is performe. Finally in Figures 9() an 9() the two vieo objets are extrate (foregroun an bakgroun). Extration of the vieo objets of Figures 5() an (e) are also presente in Figures 10 an 11 respetively, where in 10(b), 11(b) projetion of the olor segments masks onto the epth maps is performe an in Figures 10(), 11() fusion of olor segments belonging to the same epth is aomplishe. Finally the extrate foregroun an bakgroun objets for the two ases an be seen in Figures 10(), 10(e) an 11(), 11(e) respetively. As it an be observe, the results are very promising even in ases of ompliate bakgrouns or of ifferent numbers of vieo objets. Furthermore it shoul be mentione that in ase of Figure 5(e) extrate vieo objets have not been very aurately etete ue to inaurate epth segmentation results, an issue whih is also evient at the upper left part of the man-objet (Figure 10()). As we will see next, this issue has greater impat in the segmentation fusion approah, sine final segmentation heavily epens on the estimate epth segments. 5.3 Segmentation Results of the Ative Contour Approah Aoring to the seon approah an ative ontour is initialize on the bounary of eah epth segment to perform VOP segmentation. Initialization is aomplishe using the propose unsupervise tehnique esribe in subsetion 4.1, where firstly an orer of the etete bounary points of a epth segment is estimate an then a fitness funtion is inorporate so that only a small portion of these points is kept. In the polygonal approximation sheme an after several experiments f max was set equal to 5 an D max equal to 170 in orer to avoi large gaps in the initial ative ontour. Then ative ontour segmentation results in ase of Figure 5(a) are presente in Figure 12. In partiular the epth ontour of the foregroun vieo objet was omprise of 794 points, but only 14 points were use to form the initial ative ontour, whih is epite in Figure 12(a). Next eges were estimate for the visual ontent Vs 1 efine by the foregroun epth segment an the resulting ege map was filtere (so that isolate eges were isare) to proue e init_f. Afterwars eah point of the initial ative ontour was assoiate to an attrative ege of e init_f as esribe in subsetion 4.1, where T num an thr were experimentally set equal to 4 an 11 respetively. The filtere ege map e init_f is epite in Figure 12(b), where initial points are skethe as green squares while attrative eges are marke as re irles. As it an be seen only 13 out of the 14 points are assoiate to an attrative ege an the remaining point moves only aoring to the internal energy E int. Afterwars the greey approah is ativate so that the initial ative ontour onverges to its final position. In ase of multiple vieo objets, onvergene of the multiple ative ontours is fully parallele, sine eah ative ontour moves inepenently. In Figure 12() vieo objet etetion is aomplishe after 57 iterations. In this ase an sine the existing epth segments are two (foregroun - bakgroun), one ative ontour is enough. Finally the two extrate vieo objets are shown in Figures 12() an 12(e) respetively. Similar results for Figures 5() an 5(e) an be seen in Figures 13 an 14 respetively. In partiular in ase of Figure 5() an for the left vieo objet (man) 918 points where etete 15 of whih were kept to omprise the initial ative ontour, while in ase of the right vieo objet (woman) 867 points were foun 21

14 of whih were kept. On the other han onerning vieo objet of Figure 5(e) 823 points were foun, 20 of whih where finally kept, after the polygonal approximation metho, to form the initial ative ontour. Initial ative ontours for Figures 5() an 5(e) an be seen in Figures 13(a) (both vieo objets) an 14(a) respetively, while the ege maps e init_f ontaining initial points an attrative eges are shown in Figures 13(b) an 14(b). As it an be observe in Figure 13(b) only 10 out of the 15 points are assoiate to an attrative ege for the left vieo objet, while in ase of the right vieo objet 20 out of 21 points have an attrative ege. In ase of Figure 14(b) all 20 points have their respetive attrative ege, two points of whih are assoiate to the same attrative ege. Finally ative ontours onvergene after 77 iterations for the left objet an 64 iterations for the right objet is presente in Figure 13(). Furthermore vieo objets extration an be seen in Figures 13() an 13(e). Similarly in ase of Figure 5(e) an after 59 iterations, the ative ontour onverges to its minimal energy position epite in Figure 14(). Finally the foregroun an bakgroun vieo objets are epite in Figures 14() an 14(e) respetively. As observe only a small perentage of points (~2% on average) is onsiere in all ases, for onstrution of the initial ative ontours, leaing to a substantial reution of the omputational omplexity. Furthermore the ative ontour sheme generally provies a goo segmentation quality even in ases of multiple vieo objets or omplex visual ontent. On the other han the problem of inaurate epth segmentation is better aresse by this tehnique, ompare to the segmentation fusion approah. 5.4 Comparison of the Two Unsupervise Vieo Objet Segmentation Shemes In this subsetion omparison of the propose vieo objet segmentation shemes is performe in terms of omputational omplexity an auray. Consiering auray, the first approahes for objetive evaluation of vieo objet segmentation results an be foun uring stanarization of the MPEG-4 [24], within the ore-experiment on automati segmentation of moving objets. The objetive evaluation sheme in [25] uses an a-priori known 2-D shape in orer to appraise the estimation result, while in [26] aitional geometri features like size an position of a vieo objet, as well as the average olor within a VOP area are also onsiere. The aforementione approahes o not istinguish between a lot of small eviations between the estimate an original mask an a few but larger eviations. Both ases an lea to the same value of spatial auray, although they are visually ifferent. In orer to onfront this problem Meh an Marques have propose a spatial auray an temporal oherene evaluation sheme base on the mean an stanar eviation of 2-D shape estimation errors [27]. Aoring to this sheme for eah pixel i of the original ontour, istane i to the estimate vieo objet ontour is measure an the mean an stanar eviation of these istanes is alulate. Assurely, attention shoul be pai in two important issues: (a) The tehnique to fin the orresponene between pixels of the original ontour an the estimate ontour is position sensitive, thus ifferent istanes an be foun for ifferent parsing iretion of the original ontour an (b) Wrong orresponenes an thus wrong istanes an be foun, ue to wrong orientation of the assignment vetors. In our approah the sheme presente in [28] is aopte, where the quality of segmentation masks is evaluate by means of algorithmially ompute figures of merit, weighte to take into aount the visual relevane of segmentation errors. This sheme assumes the existene of a perfet mask so that evaluation an

15 ranking of the performane of segmentation algorithms an be aomplishe. For this reason a spatial quality measure (SQM) with higher pereptive meaning is buil, by onsiering that some errors are visually more important than others. Aoring to this sheme the following efinitions an assumptions are mae: - Two types of error exist: (a) missing foregroun points (MF) an (b) ae bakgroun points (AB), with ifferent visual importane. - Wrong pixels have bigger visual relevane when loate further away from the borer of the referene mask. - MF errors are usually more tolerable than AB errors at istanes very near to the referene mask, sine in the latter ase bakgroun noise is ae to the vieo objet (halo effet), while in the former ase a slightly thinne but usually aeptable VOP is extrate. - Moving away from the borer, MF errors attain greater relevane, sine a bigger part of the objet is being misse. - Although AB errors also inrease their relevane at far loations, this inrement tens to stabilize with the istane. Then the absolute spatial quality measure is efine as: D FG max SQM= 1 D BG max w MF () Car(R E ) + 1 w AB () Car( R E) (17) where E is the estimate mask, R the referene mask, ( ) iniates the omplement of a set, D FGmax an D BGmax are the maximum istanes for the MF an AB pixels respetively an sets R are efine as: R i = x xєr, (x,r )= i (18a) R i = x xєr, (x,r)= i Furthermore a iagram of the weighting funtions w MF () an w AB () is epite in Figure 16. Finally for normalization purposes the SQM value is ivie with the number of pixels of mask R. For the 4 foregroun vieo objets of Figures 5(a), 5() an 5(e) the manually reate referene masks are shown in Figures 15(a), 15(b), 15() an 15() respetively. (18b) Comparison of the two propose tehniques for the extrate foregroun vieo objets is provie in Tables I, II, III an IV. The first olumn is oupie by the perfet mask an the two estimate masks to be ompare (fusion mask, ative ontour mask). The seon olumn refers to the number of pixels that eah mask ontains. In the thir olumn, the number of pixels of eah mask that belong to the VOP area (base on the perfet mask) is presente. The fourth olumn ontains fator CP, whih expresses the perentage of the perfet vieo objet mask overe by the estimate mask an is alulate by: Number of estimate pixels of vieo objet CP (19) Number of pixels of vieo objet The fifth olumn ontains fator AP, whih expresses the perentage of the estimate mask that overs area of the vieo objet an is alulate by: Number of estimate pixels of vieo objet AP (20) Number of pixels of the estimate mask

16 The sixth olumn ontains the value of SQM while in the seventh olumn the SQM (logarithmi sale of SQM) is provie that is estimate by: 1 SQM 10 log (21) 1 SQM Finally in the eighth olumn exeution times are also provie. Exeution times were estimate for a Pentium 2.6 GHz proessor, 512 MB RAM an ANSI C implementation of the various shemes. As it an be observe from Table I, whih refers to the foregroun vieo objet of Figure 5(a) (woman), both tehniques provie very goo results with the best results given by the fusion tehnique. Here it shoul be mentione that for SQM values between 0 an -2.5 B the extrate vieo objets have highly aurate ontours. For SQM in the range of [-5 2.5) B vieo objets are generally very well etete but there are also small visually observable areas that are overe an o not belong to the vieo objet or small regions not overe an belonging to the vieo objet. Furthermore for values in the range [-7.5 5) B vieo objets are well etete but now the visually observable missing foregroun or ae bakgroun areas are larger. Finally for values less than 7.5 B segmentation auray ereases, proviing in many ases low quality results. Returning to Table I, most of the vieo objet area is overe by the estimate masks (~ 95.2 % on average) while the greatest part is overe by the mask of the fusion sheme (95.5 %). On the other han eah of the estimate masks mainly overs vieo objet area (~ 97.5 % on average). Finally SQM is worse for the mask estimate by the ative ontour sheme. This is ue to the fat that there are no eges at the lower borer of the vieo objet uner onsieration. On the other han in terms of time the ative ontour approah is faster ahieving segmentation time of 0.55 se against 1.06 se of the fusion sheme, leaing in about 48% reution of segmentation time ompare to the fusion sheme. In Table II results for the left foregroun vieo objet of Figure 5() (man), are presente. In partiular in this ase both the ative ontour an CFCS approahes provie similar SQM values. This is ue to inaurate epth segmentation, as the upper left part of the image is ontaine in the same epth segment with the vieo objet. In terms of time, the ative ontour approah is faster giving an improvement of about 61 % ompare to the fusion sheme. In Table III results for the right foregroun vieo objet of Figure 5() (woman) an be observe. More speifially in this ase the segmentation fusion metho gives more aurate results (-2.33 SQM / 99.5% overage). However the ative ontour sheme is again faster with exeution time 0.69 se against 1.9 se performe by the fusion sheme (about 64 % improvement). Finally in Table IV results for the foregroun vieo objet of Figure 5(e) (ar) are shown. In partiular the ative ontour tehnique ahieves lower SQM, proviing however worse overage of the vieo objet area (93.7 %). On the other han the fusion sheme provies a very high SQM value ( B) an an AP equal to 68.9 %. Again it is evient that inaurate epth segmentation affets more the CFCS approah. For ompleteness reasons the overall performane of the CFCS an ative ontour approahes in terms of segmentation auray (SQM ) an omplexity (exeution time) are illustrate in Figures 17(a) an (b) respetively, for more than 500 vieo objets. As it an be observe the CFCS approah ahieves better auray in most ases, where aurate epth segmentation is aomplishe, proviing mean SQM =-3.92

17 while in ase of the ative ontour approah the mean SQM = On the other han the ative ontour sheme is faster than the CFCS sheme (Figure 17(b)), proviing an average exeution time equal to 0.94 se (1.93 se for the CFCS approah), whih leas to a omplexity improvement of about 51 %. 6. Conlusions In this paper two epth-base unsupervise shemes for VOP segmentation have been propose that an omplement several existing methos base on motion information. Both shemes reeive as input stereosopi sequenes, proviing MPEG-4 ompatible 2-D sequenes in their output. In partiular in both shemes stereosopi pairs are initially analyze an epth segments maps are proue by applying a moifie version of the M-RSST segmentation algorithm. Afterwars the first sheme (CFCS) performs VOP segmentation by appropriately ombining olor an epth esriptors in a hierarhial framework. Towars this iretion one of the stereosopi hannels is partitione into olor segments by utilizing the onventional M-RSST algorithm. Color segments provie very aurate bounaries but oversegment a VOP into multiple regions. On the other han epth segments effiiently esribe semanti visual ontent, but usually blur VOPs' bounaries. For this reason, the propose CFCS algorithm merges olor segments base on epth onstraints. The main issue of CFCS is its omputational ost, whih is worse ompare to the ative ontour tehnique, while its main avantage is its very goo performane in terms of segmentation quality. On the other han the ative ontour sheme is base on the iea of energy minimization. An ative ontour is initialize onto the bounary of eah epth segment by utilizing information provie by the epth ontour. Initialization is unsupervisely performe, in ontrast to most existing supervise shemes. In the next phase, an sine usually epth segments bounaries are near to the VOPs bounaries, eah point of the ative ontour is assoiate to an attrative ege point aoring to istane an olor riteria, so that the ative ontour evolves faster an reliably towars the bounary of the VOP. Finally an effiient an osteffetive greey approah is inorporate uring onvergene. Both shemes provie promising results an an support a new range of apabilities in terms of aess, reation, manipulation or eiting of visual ontent. In partiular, ontent segmentation enables for more effetive image/vieo oing, permits sophistiate ontent-base queries on multimeia atabases an enables oers to perform rate ontrol aoring to semanti information. Furthermore, although the propose algorithms are esigne for unsupervise ontent segmentation, they an also be inlue in interative appliations by making minor moifiations, resulting in semi-automati shemes that woul provie more aurate segmentation results. Finally, both approahes are not limite to exploiting epth information but an also use any kin of information that roughly esribes a semanti objet, base eah time on the speifi appliation. 7. Aknowlegments The authors wish to thank very muh Dr. Chas Girwoo, the projet manager of the ITC (Winhester), for proviing the 3D vieo sequene Eye to Eye, whih was proue in the framework of ACTS MIRAGE projet. Furthermore the authors want to thank very muh Dr. Siegmun Pastoor of the HHI

18 (Berlin), for proviing the vieo sequenes of the DISTIMA projet. This work is supporte by the EU projet MESH Multimeia Semanti Syniation for Enhane News Servies (FP ). 8. Referenes [1] ISO/IEC JTC1/SC29/WG11, MPEG-4 Visual Final Committee Draft, Dublin, Irelan, July [2] B. Furht, S.W. Smoliar an H. Zhang, Vieo an Image Proessing in Multimeia Systems. Κluwer Aaemi Publishers, [3] N. Doulamis, A. Doulamis, Y. Avrithis, K. Ntalianis an S. Kollias, Effiient Summarization of Stereosopi Vieo Sequenes, IEEE Trans. Ciruits an Systems for Vieo Tehnology, Vol. 10, No. 4, pp , June [4] R. Castagno, T.Ebrahimi, an M. Kunt, Vieo Segmentation Base on Multiple Features for Interative Multimeia Appliations, IEEE Trans. Ciruits an Systems for Vieo Tehnology, Vol. 8, No. 5, pp , [5] C. Gu an M.-C.Lee, Semiautomati Segmentation an Traking of Semanti Vieo Objets, IEEE Tran. Ciruits an Systems for Vieo Tehnology, Vol. 8, No. 5, pp , [6] C. Xu an J. L. Prine, Snakes, Shapes an Graient Vetor Flow, IEEE Trans. Image Proessing, Vol. 7, No. 3, pp , Marh [7] S. Lobregt an M. A. Viergever, A isrete ynami ontour moel, IEEE Trans. Meial Imaging, Vol. 14, pp , Marh [8] D. Wang, Unsupervise Vieo Segmentation Base on Watershes an Temporal Traking, IEEE Tran. Ciruits an Systems for Vieo Tehnology, Vol. 8, No. 5, pp , [9] T. Meier, an K. Ngan, Vieo Segmentation for Content-Base Coing, IEEE Tran. Ciuits an Systems for Vieo Tehnology, Vol. 9, No. 8, pp , [10] M. Kim, J.G. Choi, D. Kim, H. Lee, M.H. Lee, C. Ahn, an Y-S. Ho, A VOP Generation Tool: Automati Segmentation of Moving Objets in Image Sequenes Base on Spatio-Temporal Information, IEEE Tran. Ciruits an Systems for Vieo Tehnology, Vol. 9, No. 8, pp , [11] H-L. Eng, an K-K. Ma, Spatio-Temporal Segmentation of Moving Vieo Objets over MPEG Compresse Domain, in Pro. of the International Conferene on Multimeia an Expo (ICME), New York, USA, August [12] R. C. Gonzalez an R. E. Woos, Digital Image Proessing. Aison-Wesley, [13] D. J. Luenberger, Linear an non-linear Programming. Aison-Wesley [14] A. D. Doulamis, N. D. Doulamis, K. S. Ntalianis, an S. D. Kollias, Unsupervise Semanti Objet Segmentation of Stereosopi Vieo Sequenes, in Pro. of the IEEE International Conferene on Information, Intelligene an Systems (ICIIS), Washington D.C., U.S.A, November [15] N. Grammaliis an M. G. Strintzis, Disparity an Olusion Estimation in Multioular Systems an Their Coing for the Communiation of Multiview Image Sequenes, IEEE Trans. Ciruits an Systems for Vieo Tehnology, Vol. 8, No. 3, pp , June 1998.

19 [16] Y. Avrithis, A. Doulamis, N. Doulamis an S. Kollias, A Stohasti Framework for Optimal Key Frame Extration from MPEG Vieo Databases, Computer Vision an Image Unerstaning, Aaemi Press, Vol. 75, Nos 1/2, pp. 3-24, July/August [17] M. Kass, A. Witkin, an D. Terzopoulos, Snakes: Ative ontour moels, International Journal of Computer Vision, vol. 1, pp ,1987. [18] L. D. Cohen an I. Cohen, Finite-element methos for ative ontour moels an balloons for 2-D an 3-D images, IEEE Transations on Pattern Analysis an Mahine Intelligene, vol. 15, pp , Nov [19] A. A. Amini, S. Tehrani, an T. E. Weymouth, Using Dynami Programming for Minimizing the Energy of Ative Contours in the Presene of Har Constraints, in Pro. of the Seon International Conferene on Computer Vision (ICCV), 1988, pp [20] D. J. Williams an M. Shah, A fast algorithm for ative ontours an urvature estimation, GVGIP: Image Unerstaning, vol. 55, no.1, pp , January [21] J. Slater, Eye to Eye with Stereosopi TV, Image Tehnology, p. 23, Nov./De [22] C. Girwoo an P. Chiwy, MIRAGE: An ACTS Projet in Virtual Proution an Stereosopy, IBC Conferene Publiation, No. 428, pp , Sept [23] M. Ziegler, Digital Stereosopi Imaging an Appliations: A Way Towars New Dimensions, the RACE II Projet DISTIMA, Pro. of IEE Colloquium on Stereosopi Television, Lonon, UK, [24] MPEG-4: Do. ISO/IEC JTC1/SC29/WG11 N2502, Information Tehnology Generi Coing of Auiovisual Objets, Part 2: Visual, Final Draft of International Stanar, Otober [25] M. Wollborn, R. Meh, Proeure for Objetive Evaluation of VOP Generation Algorithms, Do. ISO/IEC JTC1/SC29/WG11 MPEG97/2704, Fribourg, Switzerlan, Otober [26] P. Correia an F. Pereira, Objetive Evaluation of Relative Segmentation Quality, in Pro. International Conferene on Image Proessing (ICIP), Vanouver, Canaa, September [27] R. Meh an F. Marques, Objetive Evaluation Criteria for 2D-Shape Estimation Results of Moving Objets, in Pro. of Workshop on Image Analysis for Multimeia Interative Servies (WIAMIS), Tampere, Finlan, May [28] P. Villegas, X. Marihal an A. Saleo, Objetive Evaluation of Segmentation Masks in Vieo Sequenes, in Pro. Of Workshop on Image Analysis for Multimeia Interative Servies (WIAMIS), Berlin, Germany, May-June Figure 1 Figure 2 Figure 3 List of Figures The propose blok iagram of the metho for prouing the epth segments map. Graphial example of the propose CFCS approah esribing the ase of a shuttle. (a) Color segments mask, (b) Depth segments map, () Projetion of olor segments mask onto epth segments map an fusion an () The final vieo objets segmentation mask. Angle formulation for estimating the bening term of the ative ontour.

20 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 Figure 14 Graphial representation of the ative ontour polygonal approximation metho. Stereosopi pairs of frames use for experimental evaluation. (a), (), (e) Left hannels an (b), (), (f) Right hannels. Disparity an epth estimation for the three stereosopi pairs. (a), (), (e) Olusion ompensate isparity fiels an (b), (), (f) The respetive epth maps. Estimation of epth segments maps. (a), (), (e) Depth segments maps proue by the moifie M-RSST algorithm (b), (), (f) Depth segments maps proue by the onventional M-RSST algorithm. Demonstration of M-RSST algorithm for olor segmentation in ase of Figure 5(a). (a) Segmentation at resolution level 3, (b) Segment splitting at level 3, () Segmentation at resolution level 2, () Segment splitting at level 2 an (e) Segmentation at resolution level 1 (final segmentation). Vieo objets extration results by CFCS approah in ase of stereosopi pair of Figures 5(a)-(b). (a) Depth segmentation overlai with olor segment ontours (in white), (b) Fusion of olor segments belonging to same epth segment, () Foregroun vieo objet an () Bakgroun vieo objet. Vieo objets extration results by CFCS approah in ase of stereosopi pair of Figures 5()-(). (a) Color segments mask, (b) Depth segmentation overlai with olor segment ontours (in white), () Fusion of olor segments belonging to same epth segment, () Foregroun vieo objets an (e) Bakgroun vieo objet. Vieo objets extration results by CFCS approah in ase of stereosopi pair of Figures 5(e)-(f). (a) Color segments mask, (b) Depth segmentation overlai with olor segment ontours (in white), () Fusion of olor segments belonging to same epth segment, () Foregroun vieo objet an (e) Bakgroun vieo objet. Vieo objets extration results by ative ontour approah in ase of stereosopi pair of Figures 5(a)-(b). (a) Ative ontour initialize onto epth bounary, (b) Initial ative ontour points (green squares) an respetive attrative ege points (re irles) epite insie e init_f, () Convergene of ative ontour to minimum energy position, () Foregroun vieo objet an (e) Bakgroun vieo objet. Vieo objets extration results by ative ontour approah in ase of stereosopi pair of Figures 5()-(). (a) Ative ontours initialize onto epth bounaries, (b) Initial ative ontours points (green squares) an respetive attrative ege points (re irles) epite insie e init_f, () Convergene of ative ontours to minimum energy positions, () Foregroun vieo objets an (e) Bakgroun vieo objet. Vieo objets extration results by ative ontour approah in ase of stereosopi pair of Figures 5(e)-(f). (a) Ative ontour initialize onto epth bounary, (b) Initial ative ontour points (green squares) an respetive attrative ege points (re irles) epite insie

21 e init_f, () Convergene of ative ontour to minimum energy position, () Foregroun vieo objet an (e) Bakgroun vieo objet. Figure 15 Manually reate referene masks use for quality evaluation of the propose unsupervise segmentation shemes. (a) Referene mask for vieo objet of Figure 5(a), (b) Referene mask for left vieo objet of Figure 5() (man), () Referene mask for right vieo objet of Figure 5() (woman) an () Referene mask for vieo objet of Figure 5(e). Figure 16 Diagram of the weighting funtions w MF () an w AB () of Equation (17). Figure 17 (a) SQM for more than 500 vieo objets an for both approahes an (b) The respetive omplexity in seons.

22 Figure 1 M A A B N C E O a b F D G H I J K L P (a) M (b) A B N C E O F D G H I J K L P () Figure 2 ()

23 y y Case 1 ω Pi ε2 ε1 Case 2 ω Pi ε1 ε2 0 Pi-1 ω1 Pi+1 ω2 x 0 Pi+1 ω2 π-ω1 Pi-1 ω1 x Figure 3 Figure 4

24 (a) (b) () () (e) (f) Figure 5

25 (a) (b) () () (e) (f) Figure 6

26 (a) (b) () () (e) (f) Figure 7

27 (a) (b) () () (e) Figure 8

28 (a) (b) () Figure 9 ()

29 (a) (b) () () (e) Figure 10

30 (a) (b) () () Figure 11 (e)

31 (a) (b) () () (e) Figure 12

32 (a) (b) () () (e) Figure 13

33 (a) (b) () () (e) Figure 14

34 (a) (b) () Figure 15 () wmf(x) wab(x) Weight Distane to the mask borer Figure 16 TABLE I 1st CASE (WOMAN) # of Pixels # of VOP Pixels CP % AP % SQM SQM' Exeution Time (se) Perfet Mask Fusion Mask Ative Contour Mask

35 TABLE II 2n CASE (MAN) # of Pixels # of VOP Pixels CP % AP % SQM SQM' Exeution Time (se) Perfet Mask Fusion Mask Ative Contour Mask TABLE III 2n CASE (WOMAN) # of Pixels # of VOP Pixels CP % AP % SQM SQM' Exeution Time (se) Perfet Mask Fusion Mask Ative Contour Mask TABLE IV 3r CASE (CAR) # of Pixels # of VOP Pixels CP % AP % SQM SQM' Exeution Time (se) Perfet Mask Fusion Mask Ative Contour Mask Figure 17(a) Figure 17(b)