A Bayesian Approach to Video Object Segmentation via Merging 3D Watershed Volumes

Transcription

1 A Bayesian Approach o Video Objec Segmenaion via Merging 3D Waershed Volumes Yu-Pao Tsai 1,3, Chih-Chuan Lai 1,2, Yi-Ping Hung 1,2, and Zen-Chung Shih 3 1 Insiue of Informaion Science, Academia Sinica, Taipei, Taiwan, R.O.C. 2 Deparmen of Compuer Science and Informaion Engineering, Naional Taiwan Universiy, Taipei, Taiwan, R.O.C. 3 Deparmen of Compuer and Informaion Science, Naional Chiao Tung Universiy, Hsinchu, Taiwan, R.O.C. Absrac In his paper, we propose a Bayesian approach o video objec segmenaion. Our mehod consiss of wo sages. In he firs sage, we pariion he video daa ino a se of 3D waershed volumes, where each waershed volume is a series of corresponding 2D image regions. These 2D image regions are obained by applying o each image frame he marker-conrolled waershed segmenaion, where he markers are exraced by firs generaing a se of iniial markers via emporal racking and hen refining he markers wih wo shrinking schemes: he ieraive adapive erosion and he verificaion agains a pre-simplified waershed segmenaion. Nex, in he second sage, we use a Markov random field o model he spaio-emporal relaionship among he 3D waershed volumes ha are obained from he firs sage. Then, he desired video objecs can be exraced by merging waershed volumes having similar moion characerisics wihin a Bayesian framework. A major advanage of his mehod is ha i can ake ino accoun he global moion informaion conained in each waershed volume. Our experimens have shown ha he proposed mehod has poenial for exracing moving objecs from a video sequence. Index Terms Video Objec Segmenaion, Waershed Segmenaion, 3D Waershed Volume, Markov Random Field. V I. INTRODUCTION ideo objec segmenaion plays an imporan role in many advanced video applicaions (such as in MPEG-4 [2] or in virual realiy), bu sill remains a challenging research opic. A popular approach o [21] video objec segmenaion is o combine a echnique for single image segmenaion wih a emporal racking procedure. Unforunaely, single image segmenaion is iself a very difficul problem (which may no be easier han video objec segmenaion). Oher echniques in [14][17] consider video sequences o be 3D signals and exend 2D mehods o hem, alhough he ime axis does no play he same role as he spaial axis. The drawback of his echnique is ha a moving objec in one frame mus overlap wih is corresponding objec in he nex frame. If he moion disance of he objec is large, he objec may become disconneced from one frame o he nex. Mos of he unsupervised segmenaion algorihms only uilize low-level feaures such as color, exure, moion, frame difference and hisogram [10][21]. However, wihou high-order informaion, semanic video objec exracion is hard o achieve. Therefore, many researches have allowed a cerain degree of human ineracion. For example, he mehods inroduced in [3][5] require some human ineracion for he iniial segmenaion of he firs image in he video. In fac, almos all he auomaic algorihms developed for exracing video objecs have some limiaions. For example, he auomaic mehod proposed in [21] can only exrac homogeneous regions, insead of complee objecs. Realizing ha here exiss no generic auomaic algorihm applicable o all kinds of video sequences, we focus on he problem of exracing video objecs having similar moion characerisic. The mehod proposed in his paper consiss of wo sages: (1) generaion of 3D waershed volumes, and (2) Bayesian merging of 3D waershed volumes. Deails of he wo sages will be 1

2 described in Secion II and Secion III. Experimenal resuls will be shown in Secion IV, and he conclusion will be given in Secion V. II. GENERATION OF 3D WATERSHED VOLUMES Waershed algorihm has been become popular echnique for image segmenaion [6][17][19]. Given a video clip, {I, 0 T}, we can regard he daa as one volume image. Our mehod firs pariions he volume image ino a se of 3D waershed volumes, where each 3D waershed volume is a series of corresponding 2D image regions. Fig. 1 shows he flowchar of our mehod for generaing 3D waershed volumes. These 2D image regions are obained by applying o each image frame he marker-conrolled waershed segmenaion described in Sep 2 of Secion II-B. The procedure for generaing 3D waershed volumes can be divided ino wo phases: iniial segmenaion and emporal racking. Deails of hese wo phases are described below. A. Iniial Segmenaion In he iniial phase, he firs frame of he video clip, I 0, is pariioned ino a se of 2D regions by applying he waershed segmenaion algorihm o he gradien image of I 0. However, he basic waershed ransformaion ends o produce over-segmenaion due o noise or local irregulariies in he gradien image. Since overly segmened regions may no be reliable enough for he nex phase of emporal racking, we adop a pre-processing mehod called opographic simplificaion o alleviae he over-segmenaion problem. In our curren implemenaion, he opographic surface of he gradien image is simplified by removing he local minima [20]. Firs, we apply a dilaion operaion wih a srucuring elemen of 2x2 pixels, i.e., le g 1 = Grad(I 0 ) B 2 2. Nex, we apply o Grad(I 0 ) a reconsrucion by erosion [18] from g 1 +h, i.e., le g 2 = φ (rec) [g 1 +h, Grad(I 0 )]. Noice ha using a larger h can eliminae more local minima. Finally, we can obain a reasonable segmenaion of I 0 by applying he basic waershed ransformaion o he simplified gradien image, g 2. In his paper, he above procedure of opographic simplificaion followed by waershed ransformaion will be referred o as he pre-simplified waershed segmenaion, and will be applied again o each subsequen frame for he purpose of refining he exraced markers, as described in Sep 1.3. Afer pre-simplified waershed segmenaion, merging of a foreground region and a background region may occasionally occur. Tha means he volume of parameer h is oo large so ha waershed regions are over-simplified. The user can selec eiher a smaller h or apply some human inervenion suppored by our sysem. Our ool allows he user o draw differen markers on some pars of he region o indicae ha hey should no be merged. Then, he marker-conrolled waershed segmenaion will be performed so ha he merged region will be spli auomaically. The operaion is qui simple for he users, and his operaion, if needed, usually is required only for he firs frame. Fig. 2 shows an example of our user inervenion ool. The edge of he ha and he background are separaed by drawing differen markers on each of hem, as shown in Fig. 2(a). Fig. 2(b) and 2(c) show he resuls afer user inervenion. B. Temporal Tracking In he second phase, our algorihm repeas he following wo seps for each subsequen frame in he video clip: (i) marker exracion, (ii) marker-conrolled waershed segmenaion. The ask of marker exracion is o exrac reliable seed regions based on he segmened regions obained from he previous frame. Given hese reliable markers, he marker-conrolled waershed segmenaion can no only accuraely exrac he boundaries of he waershed regions, bu also can deec newly emerging regions. Sep1: Marker Exracion Marker exracion is crucial o he success of he emporal racking phase and deserves some special aenion here. Our mehod 2

3 for exracing markers consiss of he following hree sub-seps: Sep 1.1: Region label propagaion by moion-based backward projecion Firs, iniial markers are obained by using backward pixel projecion based on backward moion vecors. Tha is, for each pixel p in he curren frame, we assign o he region label of he corresponding pixel in he previous frame o i. The correspondence is deermined by using he backward moion vecor m p. Here, we choose o use backward moion o avoid generaing empy and conflicing areas in he curren frame. The dense field of backward moion vecors is esimaed by using a emplae-maching algorihm ha adops adapive windows, similar o he one used in [7]. To save he compuaion ime, we firs esimae a sparse field of moion vecors a every 4 4 pixel spacing. Then, he dense pixel-wise moion vecors are compued using bilinear inerpolaion. The approximaion error can be deal wih he following process. Sep 1.2: Removing unreliable pixels from iniial markers by ieraive adapive erosion Since moion vecors are usually no very accurae, we mus remove unreliable region assignmens due o erroneous pixel correspondences. In order o reduce he possibiliy of generaing false boundaries in he nex sub-sep, he exraced markers should be as large as possible, and compleely conained in heir rue corresponding regions - which are unforunaely unknown o he compuer. if Consider an iniial marker M i. A pixel p M i, is regarded as an unreliable pixel if i has an unreliable region propagaion, ha is, ε ( p) is greaer han k E i, where ( p) exure, including inensiy and color, beween he corresponding pixels): ε denoes he local mean of exural error cenered round pixel p (ha is, he error of 1 ε ( p) = I ( p ) I 1( p + m p ) (1) NUp p U p where U p = { p and is 8-neighbors having he same region label as p}, NU p is he number of elemens in he se Up, and Ei denoes he global mean of exural error for he whole area of marker M i : where NM i 1 E i = min max + I ( p ) I 1( p m p ), 2, 16 (2) N Mi p M i is he number of he pixels in marker Mi. The reason for consraining E i o 2 and 16 is o preven using an unreasonable large or unreasonable small hreshold. The wo number, 2 and 16, are deermined according o our experimens. In his sub-sep, we apply an ieraive adapive erosion o rim off unreliable border pixels of he iniial markers, as illusraed in Fig. 3. The adapive erosion (erode if unreliable ) is performed ieraively wih a cross-shaped srucuring elemen of 5 pixels, shown in Fig. 3(b), unil he resul becomes sable. Noice ha he adapively eroded marker shown in Fig. 3(e) is a union of he normally eroded marker (shown in Fig. 3 (d)) and he reliable pixels, coloured in whie, are conained in he border porions (shown in Fig. 3(c)). Noe ha using a lower k can eliminae more marker pixels. In he case of foreground and background objecs, which are no disincive, k should be se conservaively. We found ha k = 1.2 works well for mos MPEG-4 es sequences in hand. The resuling markers wih differen values of k using frame 116 of he foreman sequence are shown in Fig. 4. Pixels in black represen any undefined areas. Sep 1.3: Removing unreliable pixels by checking wih a pre-simplified waershed segmenaion Here, we firs generaed a reasonably fine segmenaion of he curren frame by applying he pre-simplified waershed 3

4 segmenaion described in Secion II-A, wih a small value of parameer h. For each generaed waershed region, check if i conains only one marker and he sole marker occupies more han half of he waershed region. If so, he sole major marker will be reained for driving he marker-conrolled waershed segmenaion in he nex sep. Oherwise, he marker pixel in his waershed region will be considered unreliable, and will be removed from he markers, as illusraed in Fig. 5. Fig. 6 shows he final markers obained by applying his sub-sep o he markers shown on Fig. 4. We can see ha afer his sep, small and ambiguous pieces of he marker are removed. Sep 2. Marker-conrolled waershed segmenaion Based on he reliable markers obained from he las sep, we can hen exrac more precise region boundaries by using he marker-conrolled waershed segmenaion [9][21]. One problem accompanying marker-conrolled segmenaion is ha no newly exposed regions can be exraced wihou creaing new markers. To solve his problem, we modify he marker-conrolled waershed algorihm slighly. For he flooding process of he marker-conrolled waershed algorihm used in [21], when he waer coming from wo differen basins is abou o mee, he wo basins are merged if boh have he same label or a leas one of hem is unlabeled. Our modificaion for creaing new markers is if he dynamics of an unlabeled basin larger han a cerain hreshold [11][8], he basin will be given a new label (Fig. 7). Fig. 8 shows he resul of deecing new regions using frame 26 and 27 of he coasguard sequence. The big boa is enering he image from he lef, and he background waer can be deeced as a new region. III. BAYESIAN MERGING OF WATERSHED VOLUMES Once he 3D waershed volumes are generaed, as described in Secion 2, we need o merge hem ino a se of desired video objecs. Here, we propose a Bayesian approach o merging waershed volumes having similar moion characerisics, hoping ha more global moion informaion can be uilized wihin a formal framework. Here, we use a Markov random field (MRF) o model he spaial and emporal relaionships among differen waershed volumes. A closely relaed work is he one done by Gelgon and Bouhemy, which uses region-level MRFs o rack a spaial image pariion [4]. Anoher work proposed by Paras e al. [14] labels waershed segmens by MAP. The labeling crierion is he maximizaion of he condiional a poseriori probabiliy of he labeling field given he moion hypohesis, he esimae of he label field of he previous frame, and image inensiies. However, our mehod is differen from heirs, no only in how he MRF is applied (we employ he MRF afer racking while hey do i before racking), bu also in how he class-condiional probabiliy is modeled. A. Exracion of Feaures from 3D Waershed Volume Before applying he Bayesian merging o 3D waershed volumes, he represenaive feaures for each waershed volume need o be exraced. Moion informaion is an imporan cue o produce semanic objecs. Hence, for each waershed volume v, we consruc a feaure vecor θ v based on moion informaion. We firs decompose each waershed volume v ino a se of regions { R 0 () v () v T} v b e, where denoes a region which can be obained by inersecing frame wih he waershed volume R v v, b(v) and e (v) are he indices of he beginning frame and he ending frame of he waershed volume v, respecively. Noe ha he indices of he beginning and ending frames of he waershed volumes can vary for he waershed volume v due o he appearance or disappearance of objecs in he scene. In pracical siuaions, image moion of a rigid objec can be approximaely modeled by a small number of moion parameers. If wo regions roughly correspond o he same 3D rigid objec, he moion parameer should be abou he same. From he above observaion, we compue a moion parameer vecor θ v for each region R v by applying he Leas-Median Squares (LMedS) robus esimaor [15] o he backward dense moion field obained from Sep 1.1 of Secion II-B. The moion parameers can be 4

5 esimaed by θˆ v = arg min median m ( ; p u p θ v ) θ v p Rv (3) where u(.) is a parameerized moion field, is defined as wo-norm, and m p is he moion vecor of pixel p in frame. Afer he parameers for all he regions in he waershed volume v are deermined, we can consruc a moion feaure vecor: [ b() v, b() v + 1,..., () v θv = θv θv θ e v ]. Noice ha he dimensionaliy of θv is (e(v)- b (v)+1) d, where d is he dimension of θ. In our curren implemenaion, he moion characerisics of R v are described by a consan moion field, ha is, u( p; θ v ) = θ v, where θ v = [ m x, m y ] and m x and m y are he coordinaes of he mean moion vecor. If an objec undergoes a complex moion or deformaion, a more complex moion model, such as a six-parameer affine model or eigh-parameer quadraic model, should be used o enhance discriminaive abiliy [12]. Once a complex moion model, such as a six-parameer affine model, is adoped, he equaions presened in nex secion should be modified slighly. B. The Proposed Mehod In his work, we assume ha he number of video objecs, N, o be exraced (including he background objecs) is known. Given a se of 3D waershed volumes V = {v j, j=1,,k}, where K is he number of 3D waershed volumes, a Volume Adjacency Graph (VAG) can be consruced o express he neighborhood relaionship among 3D waershed volumes. Each node in he graph corresponds o a waershed volume, and beween wo volumes exiss an arc if he volumes are spaially conneced. Nex, we define { v } { } a label field L = l lv [1.. N], v V on he VAG. Given M = θ v v V, we esimae he labeling field L by maximizing he a poseriori probabiliy (MAP). Using he Bayes rule, he a poseriori probabiliy densiy funcion can be expressed as: P ( L M ) P( M L) P( L) (4) The firs erm on he righ-hand side of (4) is he condiional probabiliy disribuion P(M L). I is modeled as a Gaussian disribuion, which implies ha each objec should have minimum moion variance. ( ) 1 e( v) P 2 v µ v V 2σ l = b ( v) ( ) M L exp θ ( l ) where l is he mean of he parameer vecors of all waershed volumes in frame whose corresponding labels are l, µ v v funcion of he size of he video objec. v 2 (5) σ l is a The second erm on he righ-hand side of (4) is he prior probabiliy disribuion P(L), which is a regularizaion erm. To ake ino accoun he degree of adjacency beween wo waershed volumes, we direcly exend a measure of adjacency degree beween wo regions proposed in [4] o ha beween wo waershed volumes: where l v j, v k l v j, vk b( v j, vk ) = l v v + g j g j, k k is he area of he common border beween vi and v j, and g j and g k are he graviy ceners of v j and v k, respecively. We model he prior as a Gibbs disribuion. Before defining a Gibbs disribuion, we need o define he cliques. Here, only wo-sie cliques are considered and sraighforwardly obained from he arc of he VAG. Le C v be he se of all binary cliques. The Gibbs disribuion is given by (6) 5

6 where Z b is a normalizing consan and U b (L), he regularizaion poenial, is defined as 1 P( L) = exp( U b ( L) ) (7) Zb U ( L) = b( v, v ) δ ( l, l ) (8) b ( v j, vk ) C v where δ(.) is a Kronecker dela funcion. The regularizaion erm ends o favor idenical labels for wo neighboring volume sies. The maximum a poseriori probabiliy (MAP) esimae of L is obained by minimizing he following energy funcion: j k 1 e( v) 2 Lˆ = arg min ( ) θ + 2 v µ lv b( v j, vk ) δ ( lv, l ) j vk (9) L v V 2σ = b( v) ( v j, vk ) C v Energy minimizaion is performed using an ICM algorihm proposed by Besag [1], someimes also called he greedy algorihm. A each ieraion, each volume sies is visied. The label of each sie is eiher changed o he label ha yields maximal decrease of he energy funcion, or lef unchanged if no energy reducion is possible. The process sops when no more changes can be made. The iniializaion of he label L is esimaed by he K-means algorihm. The iniial cluser means { µ (l ) 1 l N} for he K-means algorihm are esimaed as follows. The firs cluser mean is he mean of he oal moion parameers. Tha is, v j vk () = T 1 µ (1) ( µ 0 (1), µ 1 (1),.., µ (1)); µ l = θv (10) N v V where N is he number of he waershed volume v inersecing wih frame. The c h -cluser mean is he feaure vecor he farhes disance from he neares cluser mean θ v ha has () v θv () v () c = 1 e () () + µ arg max min c c v V v e v b v µ θ 1 1 1= b In summary, he algorihm of our mehod for merging waershed volumes ino video objecs can be described as follows: Inpu: Volume Adjacency Graph (VAG), K 1. Obain iniial cluser means for K-means algorihm using equaions (10) and (11). 2. Obain iniial label for each waershed volume by applying K-means algorihm. 3. Updae labels for all volumes by applying ICM algorihm based on equaion (9). Oupu: Labels of all waershed volumes ( c ) (11) IV. EXPERIMENTAL RESULTS In his secion, we use he foreman, and coasguard sequences, shown in Fig. 9, and Fig. 10, respecively, o demonsrae he performance of our algorihm. In our curren implemenaion, he gradien images are compued on a weighed YUV colour space, i.e. w y Y+w u U+w v V. The weighing facors, w y, w u, and w v, are se o one, wo, and wo, respecively, o sress he color componens. The experimens are run in AMD Ahlon 1.2GHz PC wih 384MB RAM. The sizes of he foreman sequence and he coasguard sequence are 352x288 and 352x240. The oal execuion ime of he foreman sequence (100 frames) is 483sec, and he coasguard sequence (50 frames) is 131sec. For he experimenal resuls presened in his paper, no user inervenion has been used. However, one can always find some video sequences ha conain complex enough scene, such ha user inervenion may become necessary. 6

7 In he foreman sequence, he human body has a moderae moion and he camera is moving as well. I can be seen from Fig. 9(b), where cross-secions of waershed volumes are shown, ha he resuls obained by marker-conrolled emporal racking look prey good. By seing N = 2 (i.e., he number of video objecs o be exraced is 2), he waershed volumes depiced in Fig. 9(b) can be correcly merged ino wo video objecs: he foreman and he background, as shown in Fig. 9(c). In his sequence, we have found ha he similariy beween he moions of he head and he shoulder could be more easily deeced when considering a longer sequence. Therefore, our mehod can obain beer segmenaion resuls han hose obained by Moschni e al. [10]. In he coasguard sequence, he horizonal camera drif is presen while wo boas are moving wih differen velociies and direcions. Noice ha he bigger boa is enering he image from he lef, and is new emerging regions can be successfully exraced, as shown in Fig. 10(b). If we se N = 4, he proposed Bayesian mehod can pariion he video clip ino four differen objecs: he bigger boa, he smaller boa, he waer and he shore, as shown in Fig. 10(c). Compared wih he resuls using he mehod proposed by Paras e al.[14], he segmened boundaries we exraced are much closer o he objecs. V. CONCLUSION In his paper, we have proposed a new mehod for video objec segmenaion. This mehod firs pariions he video daa ino a se of 3D waershed volumes, and hen exracs video objecs by merging moion-coheren waershed volumes wihin a Bayesian framework. One major conribuion of his work is ha i models he prior informaion wih a MRF over a Volume Adjacency Graph (VAG), where each node of he VAG is a 3D waershed volume, and hence, is able o ake ino accoun he global moion informaion conained in each waershed volume. This mehod is appropriae for exracing objecs having similar moion because i can merge 3D waershed volumes having similar moion wih a Bayesian framework. Anoher conribuion is ha his paper proposes an efficien way o exrac reliable markers by shrinking wih wo schemes: he ieraive adapive erosion and he verificaion agains a pre-simplified waershed segmenaion. Experimenal resuls have shown ha he proposed mehod has poenial for exracing moving objecs from a video sequence. REFERENCES [1] J. Besag, On he saisical analysis of diry picures, J. R. Sa. Soc. B, 48(3): , [2] P. Daras, I. Kompasiaris, I. Grinias, G. Akrivas, G. Tzirias, S. Kollias and M.G. Srinzis: MPEG-4 Auhoring Tool using Moving Objec Segmenaion and Tracking in Video Shos, EURASIP J. on Applied Signal Processing, vol. 2003, no. 9, pp , Augus [3] D. Gaica-Perez, G. Gu and Ming-Ting Sun, Semanic Video Objec Exracion Using Four-Band Waershed and Pariion Laice operaors, IEEE Trans Circui Sysems Video Technology, 11(5): , [4] M. Gelgon and P. Bouhemy, A Region-Level Moion-Based Graph Represenaion and Labeling for Tracking a Spaial Image Pariion, Paern Recogniion, 30(4): , [5] C. Gu and M. Lee, Semiauomaic Segmenaion and Tracking of Semanic Video Objecs, IEEE Trans. Circui Sysems Video Technology, 8(5): , [6] K. Haris, S. N. Efsraiadis, N. Maglaveras, and A. K. Kasaggelos, Hybrid Image Segmenaion Using Waersheds and Fas Region Merging, IEEE Trans. Image Processing, 7(12): , [7] T. Kanade and M. Okuomi, A Sereo Maching Algorihm wih an Adapive Window: Theory and Experimen, IEEE Trans. Paern Analysis and Machine Inell., 16(9): , [8] C. Lemarechal, and Fjorof, Commens on Geodesic Saliency of Waershed Conours and Hierarchical Segmenaion, IEEE Trans. Paern Analysis and Machine Inell., 20(7): , [9] F. Meyer and S. Beucher, Morphological Segmenaion, J. Visual Commu. Image Represenaion, 1:21-46, [10] F. Moscheni, S. Bhaacharjee, and M. Kun, Spaioemporal Segmenaion Based on Region Merging, IEEE Trans. Paern Analysis and Machine Inell., 20(8): , [11] L. Najman, and M. Schmi, Geodesic Saliencey of Waershed Conours and hierarchical Segmenaion, IEEE Trans. Paern Analysis and Machine Inell., 18(12): , [12] H. T. Nguyen, M. Worring and A. Dev, Deecion of Moving Objecs in Video Using a Robus Moion Similariy Measure, IEEE Trans. on Image Processing, 9(1): , January

8 [13] M. Pardas, P. Salembier, 3D Morphological Segmenaion and Moion Esimaion for Image Sequences, Signal Processing, 38, pp , [14] I. Paras, E. A. Hendriks and R. L. Lagendijk, Video Segmenaion by MAP Labeling of Waershed Segmens, IEEE Trans. on Paern Analysis and Machine Inell., 23(3): , March [15] P.J. Rousseeuw, Leas Median of Squares Regression, J. Amer. Sais. Assoc., 79, pp , [16] J. B.T.M. Roerdink, and A. Meijser, The Waershed Transform: Definiions, Algorihms and Parallelizaion Sraegies, Fundamena Informaicae, 41: , [17] P. Salembier and M. Pardas, Hierarchical Morphological Segmenaion for Image Sequence Coding, IEEE Trans. Image Processing, 3(5): , Sepember [18] L. Vincen, Morphological Grayscale Reconsrucion in Image Analysis: Applicaion and Efficien Algorihms, IEEE Trans. on Image Processing, 2: , [19] L. Vincen and P. Soille, Waersheds in Digial Spaces: An Efficien Algorihm Based on Immersion Simulaions. IEEE. Trans. Paern Analysis and Machine Inell., 13(6): , [20] D. Wang, A Muliscale Gradien Algorihm for Image Segmenaion Using Waersheds. Paern Recogniion, 30(12): , [21] D. Wang, Unsupervised Video Segmenaion Based on Waersheds and Temporal Tracking, IEEE Trans. Circui Sysems Video Technology, 8(5): ,

9 Iniial Segmenaion (II.A) I 0 S 0 Temporal Tracking (II.B) (II.B Sep 1.1) I -1 I Moion Esimaion Region Label Propagaion Removing Unreliable Pixels (II.B Sep 1.2) Pre-Simplified Waershed Segmenaion (II.B Sep 1.3) Removing Unreliable Pixels Marker-Conrolled Waershed Segmenaion (II.B Sep 2) Fig. 1. Follow cha of generaing 3D waershed volumes. S (c) (d) Fig. 4. Markers exraced from frame 116 of sequence foreman wih differen he value of k afer Sep 1.2 for marker exracion. (a) k = 0.8. (b) k = 1.0. (b) k = 1.2. (b) k = 1.4. Regions where markers cover less han 50% of he area Ambiguous regions (conaining more han wo markers) (a) (b) Fig. 5. An example of Sep 1.3 for marker exracion. (a) Two differen markers are overlaid by he waershed lines obained from pre-simplified waershed segmenaion. (b) The shrunk marker afer removing he doubful porions. (a) (b) (c) Fig 2. Example of user assisance (a) (b) (c) (d) (e) (f) Fig. 3. An example of Sep 1.2 for marker exracion. (a) An iniial marker wih unreliable pixels coloured in grey. (b) A cross-shaped srucuring elemen of 5 pixels. (c) Border pixels removed wih he normal erosion. (d) Inerior pixels obained wih he normal erosion. (e) The eroded marker afer he firs ieraion of adapive erosion. (f) Afer he second ieraion (sable and sopped). (a) (c) (d) Fig. 6. Markers exraced from frame 116 of sequence foreman wih differen he value of k afer Sep 1.3 for marker exracion. (a) k = 0.8. (b) k = 1.0. (b) k = 1.2. (b) k = 1.4. (b) New region (a) (b) Marker Fig. 7. A new region is labelled if he dynamics of a cachmen basin exceeds a cerain hreshold. 9

10 (a) (b) (c) (d) Fig. 8. A demonsraion of deecing new region by using dynamic hresholding. (a) frame 26. (b) frame 27. (c) segmened resul of frame 26. (d) segmened resul of frame 27. (a) (b) (c) Fig.10. Coasguard sequence: Frame 1, 10, 20, 30, 40, 50. (a) Original Images. (b) Afer emporal racking. (c) Afer Bayesian merging. (a) (b) (c) Fig. 9. Foreman sequence: Frame 1, 20, 40, 60, 80, 100. (a) Original images. (b) Afer emporal racking. (c) Afer Bayesian merging. 10