Unsupervised Stereoscopic Video Object Segmentation Based on Active Contours and Retrainable Neural Networks

Size: px

Start display at page:

Download "Unsupervised Stereoscopic Video Object Segmentation Based on Active Contours and Retrainable Neural Networks"

Abigail Cox
5 years ago
Views:

1 Unsupervised Stereosopi Video Objet Segmentation Based on Ative Contours and Retrainable Neural Networks KLIMIS NTALIANIS, ANASTASIOS DOULAMIS, and NIKOLAOS DOULAMIS National Tehnial University of Athens Eletrial and Computer Engineering Department 9, Heroon Polytehniou str., Zografou 5773, Athens, GREECE Abstrat: - In this paper an unsupervised sheme for stereosopi video objet extration is presented based on a neural network lassifier. More partiularly, the proedure inludes: (A) A retraining algorithm for adapting neural network weights to urrent onditions and (B) An ative ontour module, whih extrats the retraining set. The retraining algorithm takes into onsideration both the former and the urrent network knowledge in order to ahieve good generalization and redue retraining time. The retrained network performs video objet traking to the rest of the frames within a shot. Retraining set extration is aomplished by utilizing depth information, provided by stereosopi video analysis and inorporating an ative ontour. Finally results are presented whih illustrate the promising performane of the proposed approah in real life experiments. Key-Words: - Adaptive neural networks, video objets, MPEG-4, depth based segmentation, ative ontours Introdution The suess of the new emerging multimedia appliations, suh as video editing, ontent-based image retrieval and video summarization, depends on the development of algorithms for effiient segmentation of the visual ontent []. Suh a ontent-based approah offers new apabilities suh as high ompression ratios by allowing the enoder to plae more emphasis on objets of interest [] and it failitates sophistiated video queries and ontentbased retrieval operations on image/video databases [3]. Towards this diretion the MPEG-4 standard introdued the onept of Video Objets (VOs), eah onsisting of arbitrarily shaped regions with different olor, texture or motion. For this reason video objet segmentation still remains a hallenging task. In stereosopi video, however, the problem of ontent-based segmentation an be effetively addressed, sine depth information is estimated more reliably and onsidering that a video objet is usually loated on a speifi depth plane [4]. On the other hand, neural networks with their superior non-linear lassifiation abilities an beome a major analysis tool in the multimediaoriented standards MPEG-4 and MPEG-7. Several tehniques and algorithms have been proposed in the literature for image segmentation. Some olor-oriented methods have been reently proposed based on the morphologial watershed [5] or by using split and merge tehniques [6]. However, an intrinsi property of video-objets is that they usually onsist of regions of totally different olor harateristis and onsequently the main problem of any olor-oriented segmentation sheme is that it oversegments an objet into multiple olor regions. Other segmentation shemes are motion-oriented, suh as the algorithms proposed in [7] and [8]. Although motion desriptors may provide a more reliable ontent representation than olor information, objet boundaries annot be identified with high auray mainly due to erroneous estimation of the motion vetors. Other semi-automati traking approahes are presented in [9], [] and []. In these methods the urrent dynami state of an objet is based on the estimation at the previous time instane. In this ase, the user initially selets an objet of interest and then the algorithm follows the objet through time, by exploiting motion information of the urve desribing the objet. In this paper, an unsupervised sheme for video objet segmentation is proposed, oriented to stereosopi video sequenes. Initially, for a given shot, the first pair of frames is analyzed and a depth

2 segments map is onstruted as in []. Then on the boundary of eah depth segment an ative ontour is automatially initialized to detet the ontained video objet. The extrated information is utilized by the retraining algorithm to optimally adjust network weights [3]. Finally the retrained network is applied to the rest of the frames within the shot, to trak the video objet of interest. Experimental results on real life stereosopi video sequenes indiate the prominent performane of the proposed sheme. Problem Formulation In our ase, video objet traking is handled as a lassifiation problem. This means that eah video objet is assigned to one lass, say ω i out of p available. Thus p is the number of video objets in an image. In this framework, eah region of the image (e.g., an image blok) is assigned to the jth objet (lass), if this region presents higher likelihood of belonging to the lass ω j, than to the other lasses ω i, i j. In this ase a neural network lassifier will produe a p-dimensional output vetor y ( x i ) defined as: i i i y( xi ) = p p... p () ω ω ω p i where p ω denotes the probability that the ith j image blok belongs to the jth lass. Let us first onsider that a neural network has been initially trained using the set S b = {( x, d ), L,( x m, d b m b )}, where vetors x i and d i with i =,, L,mb denote the ith input training vetor and the orresponding desired output vetor onsisting of p elements. Let y ( x i ) denote the network output when applied to the ith blok of an image outside the training set. Whenever a hange of the environment ours, new network weights should be estimated through a retraining proedure. Let w b be all the weights of the network before retraining, and w a the new weights obtained through retraining. A training set S is assumed to be extrated from the urrent operational situation omposed of, say, m bloks; S = {( x, d), L( xm, d )} m where x i and d i with i =,, L,m similarly orrespond to the ith input and desired output retraining data. Then the T retraining algorithm omputes the weights minimizing the following error riterion, E a E, a + ηe f, a w a, by = () m with E, a = za ( xi ) d i and i= (a) m b E f, a = z a ( x i ) d i (b) i= where E, a is the error performed over training set S, E f, a the orresponding error over training set S b ; z a ( x i ) and z a ( x i ) are the outputs of the retrained network, orresponding to input vetors x i and x i respetively, of the network onsisting of weights w a. Similarly z b ( x i ) would represent the output of the network, onsisting of weights w b. Parameter η is a weighting fator aounting for the signifiane of the urrent training set ompared to the former one. 3 The Retraining Tehnique Let us, for simpliity, onsider i) a two-lass lassifiation problem, where lasses ω, ω refer to foreground and bakground objets in an image and ii) a feedforward neural network lassifier whih inludes a single output neuron, one hidden layer onsisting of q neurons, and an input layer aepting image bloks of, say, J pixels. Then the output of the network an be expressed as [3]: u x f T (( x ) { a, ( j ) = W { a, ) j (3) where subsripts { a, refer to the states either after or before retraining, W { a, is a Jxq matrix defined as = [ w w ] W { a,,{ a,... q,{ a,, w k, { a, k=,,..,q denotes the J vetor of weights between the kth hidden neuron and the network inputs and f(.) is the hyperboli sigmoid funtion. Furthermore let us denote as w { a, a vetor ontaining the q weights between the output and hidden neurons. The goal of the training proedure is to estimate the new network weights w a, i.e., W a and w a respetively. Let us first assume that a small perturbation of the network

3 weights w b is enough to ahieve good lassifiation performane. Then, W = Wb W a + and wa w wb + w = (4) where W and are small inrements. This assumption leads to an analytial and tratable solution for estimating w, sine it permits linearization of the non-linear ativation funtion of the neuron, using a first order Taylor series expansion. To stress the importane of urrent training data in (), one an replae (a) by the onstraint that the atual network outputs are equal to the desired ones, that is a z a ( x ) = di i i =,..., m, for all data in S (5) It an be shown through linearization, that solution of (5) with respet to the weight inrements is equivalent to a set of linear equations [3] where T T w = [( w ) ( w ) ] T w = ve{ W }, with ve{ W } = A w (6) and denoting a vetor formed by staking up all olumns of W ; vetor and matrix A are appropriately expressed in terms of the previous network weights. In partiular T = [ za ( x) Lza ( xm )] [ zb ( x ) zb ( xm )] T L, expressing the differene between network outputs after and before retraining for all input vetors in S. Based on (5), vetor an be written as T [ d Ld ] [ z ( x z ( x ] T ) = m b L b m ) (7) The dimension of vetor is in general smaller than the number of unknown weights w, sine a small number of training data m is usually hosen. Uniqueness is imposed by an additional requirement due to the term E f, a in (b). In partiular, Eq. (6) is solved with the requirement of minimum degradation of the previous network behavior, i.e., of minimization of the following error riterion. ES = E f, a E f, b (8) with E f, b defined similarly to E f, a, with z a replaed by zb in the right hand side of (b). It an be shown [3] that (8) takes the form of T T ES = ( w) K K w (9) where the elements of matrix K are expressed in terms of the previous network weights w b and the training data in S b. Thus, the problem results in minimization of (9) subjet to linear onstraints (6). In this paper the problem is solved by adopting the gradient projetion method [4]. 4 Retraining Set Extration In the previous setion the retraining algorithm was presented. Thus if the retraining set is available new network weights an be diretly estimated. Retraining set extration is performed by an ative ontour module. An ative ontour is a parametri urve represented by a vetor v ( s, t) = ( x( s, t), y( s, t) ) and moving onto an image plane to minimize the energy funtional: * E snake E snake = (a) * Esnake = Einit ( v( s)) + Eimage( v( s)) ds (b) + Eon( v( s)) ds where E init represents the internal deformation energy (strething and bending), E image expresses image fores and E on orresponds to the external onstraint fores. In the proposed sheme the energy to be minimized is formulated as: * E snake = ( we int + we edge _ map ) () where E edge _ map is an external energy and E init represents the internal deformation energy. More details an be found in [5]. 4. Ative Contour Initialization and Convergene As mentioned an ative ontour is initialized onto the boundary of eah depth segment in order to extrat the video objet within the segment. More partiularly let us assume that after applying

(a) (b) () (a) (b) () (d) (e) Figure : (a) Video objet in depth (b) Initial ative ontour () Final ative ontour (d) Retraining set for the

(d) (e) Figure : (a) Video objet in depth (b) Initial ative ontour () Final ative ontour (d) Retraining set for the (a) (b) () (d) (e) (f)

stereosopi analysis methods and inorporating a segmentation algorithm, a depth segments map is produed [], omprised by Ds binary segments.

To address initialization let us denote as K the normalized differene of the image intensities between two points ( x i, yi ) and ( x i+, yi+

Let us also denote as Ν mk the minimum number of points to be kept, as Νd the total points of the depth ontour and P the perentage of seleted

Then, the following relation is held, Ν mk = Ν d Ρ Ν r = Ρ (3) From (3) it an be seen that N r is the maximum allowed number of points to be

intensity hange 3% ompared to the previous seleted point (K=3) or (b) Its previous Νr - points have been rejeted.

More partiularly eah pixel of the ative ontour starts moving on a grid in an iterative sheme, towards the diretion that minimizes the energy

The total energy is omputed for every pixel in this area and the ative ontour point moves to a new position inside the area that satisfies the

4 (a) (b) () (a) (b) () (d) (e) Figure : (a) Video objet in depth (b) Initial ative ontour () Final ative ontour (d) Retraining set for the foreground objet (e) Retraining set for the bakground objet. (d) (e) Figure : (a) Video objet in depth (b) Initial ative ontour () Final ative ontour (d) Retraining set for the foreground objet (e) Retraining set for the bakground objet. (a) (b) () (d) (e) (f) Figure 3: (a,b,) Traking results for the first shot. (d,e,f) Traking results for the seond shot. stereosopi analysis methods and inorporating a segmentation algorithm, a depth segments map is produed [], omprised by Ds binary segments. Then an ative ontour is unsupervisedly initialized onto the depth ontour of eah segment. To address initialization let us denote as K the normalized differene of the image intensities between two points ( x i, yi ) and ( x i+, yi+ ) loated on the depth ontour. Κ = Ι( xi, yi ) Ι( xi+, yi+ ) Ι( xi, yi ) () where Ι ( x i, yi ) denotes the intensity of a point. Let us also denote as Ν mk the minimum number of points to be kept, as Νd the total points of the depth ontour and P the perentage of seleted points. Then, the following relation is held, Ν mk = Ν d Ρ Ν r = Ρ (3) From (3) it an be seen that N r is the maximum allowed number of points to be rejeted between two suessive points. In our sheme a point is seleted as initial point if it validates one of the following rules: (a) It presents intensity hange 3% ompared to the previous seleted point (K=3) or (b) Its previous Νr - points have been rejeted. Then a greedy algorithm is inorporated [6], whih allows the initial ative ontour to onverge to the VOP's ontour within the depth segment. More partiularly eah pixel of the ative ontour starts moving on a grid in an iterative sheme, towards the diretion that minimizes the energy in equation (). The grid is entered at the proessed pixel and overs the 8-onneted area. The total energy is omputed for every pixel in this area and the ative ontour point moves to a new position inside the area that satisfies the onditions: The new position has less energy than the urrent position of the ative ontour pixel. The new position has the least energy ompared to the other eight pixels of the grid. If these onditions are not satisfied then the point of the ative ontour stays at its urrent position and the algorithm ontinues with the next ative ontour point. The algorithm terminates if no position hanges our during an iteration. Finally the extrated video objets onstitute the retraining set. After the network is retrained it

(a) (b) () Figure 4: The traking performane for another shot. (a) The original frames.

() The respetive traking of the seond foreground objet.

5 Experimental Results In this setion, the performane of the proposed sheme is investigated.

produed in the framework of the ACTS MIRAGE projet by AEA and ITC [7].

In partiular in (a) the foreground video objets are presented while in (b) and () the initial and

The resulting foreground and bakground retraining sets an be observed in (d) and (e) respetively.

5 (a) (b) () Figure 4: The traking performane for another shot. (a) The original frames. (b) The respetive traking of the first foreground objet. () The respetive traking of the seond foreground objet. performs traking of the video objets to the rest of the frames of a shot. 5 Experimental Results In this setion, the performane of the proposed sheme is investigated. The results have been obtained using two stereo shots taken from the stereosopi program Eye to Eye, produed in the framework of the ACTS MIRAGE projet by AEA and ITC [7]. In Figures and extration of the retraining sets for the two shots are depited. In partiular in (a) the foreground video objets are presented while in (b) and () the initial and final ative ontours are depited respetively. The resulting foreground and bakground retraining sets an be observed in (d) and (e) respetively. Finally the performane of the neural network lassifier is shown in Figure 3 for the foreground video objets of the two stereosopi shots. For presentation purposes 3 frames are presented whih are loated at different equidistant time instanes (every 5 frames) among the shots. The proposed algorithm is also evaluated using a shot with two moving objets (ators) in a very ompliated bakground. The extrated objets are presented in Figure 4. As an be seen, the traking results are very good verifying the fat that the proposed neural network arhiteture an trak video objets with auray.

6 6 Conlusions Semanti video objet extration remains a very hallenging researh problem and a generally appliable sheme has not been developed yet. In this paper the problem is onfronted by a ombination of ative ontours and a retraining proedure. In partiular an ative ontour extrats the retraining set and the neural network adapts its weights aording to this set, without forgetting the previous network knowledge. Finally the trained network is applied to the rest of the frames of a shot, in order to extrat the video objet. The tehnique is fully unsupervised and an be fully paralleled. Experimental results have been demonstrated whih indiate the promising performane of the proposed sheme. 7 Aknowledgments The authors wish to thank very muh Dr. Chas Girdwood, projet manager of the ITC (Winhester), for providing the 3D video sequene Eye to Eye, whih was produed in the framework of ACTS MIRAGE projet. Furthermore the authors want to express their gratitude to Dr. Siegmund Pastoor of the HHI (Berlin), for providing the video sequenes of the DISTIMA projet. This work is supported by the National Sholarships Foundation of Greee. Referenes: [] K. N. Ngan, S. Panhanathan, T. Sikora and M.- T. Sun, Guest Editorial: Speial Issue on Representation and Coding of Images and Video, IEEE Trans.CSVT, Vol. 8, No. 7, pp , November 998. [] T. Sikora, The MPEG-4 Video Standard Verifiation Model, IEEE Trans. CSVT, Vol. 7, No., pp. 9-3, February 997. [3] B. Furht, S.W. Smoliar and H. Zhang, Video and Image Proessing in Multimedia Systems. Κluwer Aademi Publishers, 995. [4] N. Doulamis, A. Doulamis, Y. Avrithis, K. Ntalianis and S. Kollias, Effiient Summarization of Stereosopi Video Sequenes, IEEE Trans. CSVT, Vol., No. 4, June. [5] F. Meyer and S. Beuher, Morphologial Segmentation, Journal of Visual Communiation on Image Representation, Vol., No., pp.-46, September 99. [6] M. Kunt, A. Ikonomopoulos and M. Koher, Seond Generation Image Coding Tehniques, Pro. IEEE, Vol. 73, pp , April 985. [7] W. B. Thompson and T. G. Pong, Deteting Moving Objets, Int. J. Comput. Vision, Vol. 4, pp , 99. [8] J. Wang and E. Adelson, Representing Moving Images with Layers, IEEE Trans. Image Proessing, Vol. 3, pp , Sept [9] C. Gu and M.-C.Lee, Semiautomati Segmentation and Traking of Semanti Video Objets, IEEE Tran. CSVT, Vol. 8, pp , 998. [] F. Bremond and M. Thonnat, Traking Multiple Non-rigid Objets in Video Sequenes, IEEE Trans. CSVT, Vol. 8, No. 5, pp , Sept [] K. Sethi and R. Jain, Finding Trajetories of Feature Points in a Monoular Image Sequene, IEEE Trans. PAMI, Vol.PAMI-9, No., pp , 987. [] A. Doulamis, N. Doulamis, K. Ntalianis, and S. Kollias, Effiient Unsupervised Content-Based Segmentation in Stereosopi Video Sequenes, Journal of Artifiial Intelligene Tools, World Sientifi Press, vol. 9, no., pp , June. [3] A. Doulamis, N. Doulamis, S. Kollias, On- Line Retrainable Neural Networks: Improving the Performane of Neural Networks in Image Analysis Problems, IEEE Trans. on Neural Networks, Vol., No., January. [4] D. J. Luenberger, Linear and Non-Linear Programming, Addison-Wesley, 984. [5] K. S. Ntalianis, A. D. Doulamis, N. D. Doulamis, and S. D. Kollias, An Ative Contour-Based Sheme towards More Semanti Segmentation of Stereosopi Video Sequenes, Pro. of the IEEE MELECON, Limmasol. [6] D. J. Williams and M. Shah, "A fast algorithm for ative ontours and urvature estimation," GVGIP: Image Understanding, vol. 55, no., pp. 4-6, January 99. [7] Girdwood and P. Chiwy, MIRAGE: An ACTS Projet in Virtual Prodution and Stereosopy, IBC Conferene Publiation, No. 48, pp. 55-6, Sept. 999.

New Fuzzy Object Segmentation Algorithm for Video Sequences *

New Fuzzy Object Segmentation Algorithm for Video Sequences * JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 24, 521-537 (2008) New Fuzzy Obet Segmentation Algorithm for Video Sequenes * KUO-LIANG CHUNG, SHIH-WEI YU, HSUEH-JU YEH, YONG-HUAI HUANG AND TA-JEN YAO Department