Hierarchical Region Mean-Based Image Segmentation

Transcription

1 Hieachical Region Mean-Based Image Segmentation Slawo Wesolkowski and Paul Fieguth Systems Design Engineeing Univesity of Wateloo Wateloo, Ontaio, Canada, N2L-3G1 Abstact Gibbs Random Fields (GRFs), which poduce elegant models, but which have vey poo computational speed have been widely applied to image segmentation. In contast to block-based hieachies usually constucted fo GRFs, the iegula egion-based appoach is a moe natual model in segmenting eal images. In this pape, we show that the fineto-coase egion-based hieachical egions famewok fo the well-known Potts model can be extended to non-edge based inteactions. By delibeately ovesegmenting at the fine scale, the method poceeds consevatively by avoiding the constuction of egions which staddle a egion bounday by computing egion mean diffeences. This demonstates the hieachical method is able to model egion inteactions though new genealizations at highe levels in the hieachy which epesent egions. Pomising esults ae pesented. Keywods: hieachical models, image segmentation, Makov Random Fields, Gibbs Random Fields, egionbased. 1 Intoduction A key poblem in compute vision is to distinguish between sepaate objects in an image scene. A citical step is that of image segmentation, which seeks to sepaate objects on the basis of distinct appeaance. The image segmentation pocess is dependent on two inteactive components: 1) a pixel dissimilaity citeion and 2) a famewok fo gouping simila pixels and sepaating dissimila ones. The focus of this pape is the pixel gouping algoithm. That is, given a specified dissimilaity citeion, what is an efficient and effective means of constucting goups of pixels o image segments? We conside egion-based hieachical methods based on Makov/Gibbs Random Fields [5] given the ease of constucting such models fo segmentation [7]. Many Gibbs Random Fields methods have been intoduced in ecent yeas [3, 5, 7, 9], howeve, most of these methods ae computationally slow and, theefoe, not pactical. To incease the convegence speed of the algoithm, it is necessay at some point to move away fom pocessing individual pixels to pocessing image patches o egions, which can be achieved using hieachical methods. In fineto-coase hieachical methods, the egions ae built fom the bottom up [11]. This allows egions to keep abitaily complex shapes at eve highe scales since no squae o othe stuctue is imposed fom the top as in a coase-to-fine hieachy. Theefoe, the esulting egions can natually fit the stuctues of the image being analyzed. In this pape, the main question becomes how to popely mege the small egions obtained afte the initial classic GRF algoithm has been applied to the image. In ou pevious wok [11], we extended the Potts model fom a pixel-level GRF into a hieachical GRF acceleating the andom walk but keeping the same minimization objective. Hee we extend the famewok to anothe egion meging modality. Namely, instead of compaing egions on the basis of bounday gadients we popose to use the diffeence of egion means. The hieachical egions model shaes similaities with a few othe models in the liteatue. Zhu s egion competition method [10] is simila in that it minimizes an enegy function. Howeve, it diffes consideably by fosteing competition between egions (expanding egions fom seeds and allowing egion splitting) instead of a caeful meging stategy adopted hee. Angulo s and Sea s odeed megings algoithm [1] is simila in that it ceates a hieachy of egion megings howeve it does this in a mophological and not stochastic famewok. Thei algoithm equies heuistics fo meging egions and a stopping citeion fo the algoithm. Othe hieachical methods have also been used such as coase-to-fine block-based appoaches [3, 7]; howeve, esulting images contain many block atefacts and the computational gain is minimal. Finally, Babu and Zhu [2] popose a Bayesian method which seaches the space of image segmentations to find the global optimum. They e- Poceedings of the 3d Canadian Confeence on Compute and Robot Vision (CRV 06)

2 fomulate the Swendsen-Wang algoithm fo gaphs by allowing the algoithm to split, mege o e-goup a sizeable subgaph (sets of pixels) and thus by achieving fast mixing at low tempeatues, it eliminates the slow Gibbs sampling pocedue. Although not hieachical in natue, this algoithm is simila to ous in that it allows goups of pixel labels to be flipped at any one time. The majo diffeence being the splitting of egions/subgaphs in addition to meging them. The pape is oganized as follows. The second section descibes the hieachical Gibbs Random Fields famewok. The thid section details the new egion mean-based hieachical appoach. Section fou pesents esults while the fifth section concludes the pape. 2 Bayesian Famewok The modelling poblems in this pape ae addessed fom the computational viewpoint by using Gibbs Random Fields to model the image segmentation pocess. Thee ae two pimay concens: how to define an objective function fo the optimal solution of the image segmentation, and how to find this optimal solution. Fo the pupose of this pape, the exact solution to ou segmentation poblem will be intepeted as the optimum solution to the optimization objective. In pinciple, the solution is staightfowad: simulated annealing [5] is widely used in solving Gibbs poblems; howeve, it is vey slow. The definition of a hieachical appoach poduces faste annealing. Suppose we ae given an image X with labels l on a pixel lattice L = {i, j} with dissimilaity citeion Φ( ). We will assume L has a fist ode neighbohood stuctue on a egula gid shown in Figue 1a (a second ode neighbohood stuctue would also be feasible). The enegy model is then witten as follows: U = {Φ( X i,j, X i,j+1 )δ li,j,l i,j+1 + i,j Φ( X i,j, X i+1,j )δ li,j,l i+1,j + β [ (1 δ li,j,l i,j+1 )+(1 δ li,j,l i+1,j ) ] } (1) whee β contols the elative constaints on the degee of egion cohesion and fagmentation, while δ li,j,l i,j+1 is the Konecke δ. This model opeates diectly on pixels and is theefoe a fine o pixel level model. β is usually detemined expeimentally. This is essentially a egion gowingtype model [8, 6] whee decisions to integate a pixel into the egion ae done with espect to the citeion Φ. The majo diffeence between this local GRF model and egion gowing methods is that it is non-causal. Model (1) suffes fom a slow andom walk of infomation as shown in Figue 2. This implies that only the slowest of annealing schedules will successfully convege. One (a) (b) Figue 1. Illustation of β and Φ inteactions between adjacent pixels/egions: (a) fist ode neighbohood on a egula gid fo the finest o pixel-level model, (b) egion neighbohood on an iegula gid fo highe level egion-based model. way to ovecome this limitation would be to mege adjacent egions in successive highe stages afte the anneale has conveged on the pevious fine level. This would occu only if the meging would lowe the oveall enegy. 3 Hieachical GRF Region Gouping A hieachical fine-to-coase egion-based appoach can be devised whee egion meging is an integal pat of the model. We fist efomulate model (1) in ode to define inteactions between egions: U (s) = {Φ (s), δ l,l + β (s), (1 δ l,l )} (2), R (s), whee is a egion indicato, s is the level on the hieachy, R (s) is the set of all egions, Φ (s), is the dissimilaity citeion between egions and and β (s), is the egion coupling paamete between egions and.whens =0, the fomulation coesponds to the special case of model (1) [11] indicating that Φ (s), and β(s), define elationships between all pixels (and ae non-zeo only fo adjacent pixels). Futhemoe, the neighbohood stuctue is now defined on an iegula gid as shown in Figue 1b. This model is non-local in that it opeates on egions athe than pixels howeve it is still Makovian as the infomation is peseved between levels in the hieachy. Indeed, the egion-to-egion inteactions ae cumulative local inteactions between the pixels. This model still pefoms a andom walk; howeve, the opeation is now sped-up since the label compaisons now happen on a egional, multi-pixel level athe than the single pixel inteactions of model (1) thus speeding the convegence pocess consideably. 2 Poceedings of the 3d Canadian Confeence on Compute and Robot Vision (CRV 06)

3 (a) (b) (c) Figue 2. Slow andom walk of annealing illustation in 2-D between two egions in a homogenous image (i.e., no enegy gadient) fo a two-label assignment (Ising model) at T =0. (a) Within the domain of flat enegies, the anneale pefoms a andom walk eventually finding one of the optimal endpoints (in this case a shaded o unshaded egion). Consideing that the pobability of flipping each vetex fom +1 to -1 is p o =1/2, vetex o has a 50% pobability of becoming +1. Ifit does then pixels x will be in the same situation othewise nothing changes. (b) To tansfom this image into a homogeneous patch, a andom walk of labels needs to take place. (c) Note, that the andom walk can esult in local minimum which can then be cicumvented at the next level in the hieachy. We ae able to keep the infomation fom level s to level s +1equivalent by constucting tansition equations between levels which tansfe the inteaction infomation, as well as by choosing a consevative value fo β. The tansition fo the dissimilaity citeion between two egions and at level s +1is descibed by a sum of all the individual distances between the egions in G (s) Φ (s+1), = t G (s) and G (s) : (s) Φt,t (3) whee t and t ae egion indicatos. The coupling paamete between two neighboing egions and at level s +1is witten in an analogous fashion: β (s+1), = t G (s) (s) βt,t (4) Theefoe, we now have model (2) which govens how the labelling is done at each level s togethe with between level tansition equations (3) and (4). Some moe details ae available in [11]. Given an appopiate distance metic Φ and egion coupling paamete β, the algoithm pefoms an accuate ovesegmentation of the image at the fist level by ceating a multitude of small, compact egions. Howeve, the edgebased egion fomation model still fails in scenaios whee egion gowing algoithms fail, namely, egion spilling fom one egion into the othe though a small gadient connection in an othewise high gadient bounday. This will be illustated in the esults section. 4 Region Mean-Based Model One way to mitigate the effects of edge-based egion compaison is to compae egions accoding to thei means. Computing the mean of a egion aveages all the pixel values within that egion including the edge pixels. This means that pats of a slowly vaying gadient should be meged into the coesponding adjacent egions o fom anothe thid egion. The mean-based model is exactly the same in appeaance as model (2). Howeve, the distance computation between egions ae now caied out between means. This intoduces a subtle shift in the way lage egions fom. At this level of pixel oganization edges do not matte as much as they did when foming small egion patches o blobs. Hee we would like to aggegate egions based on thei featues such as thei colo. We think that the egion mean (in this case fist pincipal component o pincipal diection of all pixels as opposed to vecto mean of pixels) will chaacteize the egions in a bette fashion and facilitate the meging of the egions that should be meged. In ode to make sue that the compaisons between the values ae simila to those in the classic Potts model the new mean-based distance is multiplied by the numbe of common edge pixels. In this way, the pevious edge-based inteaction is now somewhat smoothed. In this model, the tansitions take on a diffeent fom fo Φ (s), β (s), and Φ(s), (they emain identical to the edge-based model fo β (s), ). Given that the model no longe caes about individual pixels but goups of pixels, the tansitions need to account fo changes in the numbe of pixels in a egion, as well as the change in egion mean values. x (s) = n (s) = Φ (s) x (s 1) n (s 1) t t n (s) (5) n (s 1) t (6), =Φ(x(s),x(s) ) (7) whee at any level s, n (s) ae the numbe of pixels in a given egion and x (s) epesent the egion means. Finally, the esults of the finest-scale segmentation ae fist median filteed to emove one-pixel egions which tend to pemeate though the hieachies unde the mean model. The image segmentation algoithm is divided into two pats: a tivial image splitting pat in the fist step and a egion meging pat in subsequent iteations. 3 Poceedings of the 3d Canadian Confeence on Compute and Robot Vision (CRV 06)

4 Algoithm: Assign labels {l i,j } andomly to pixels {X i,j } Make each {X i,j } its own egion Loop ove levels: fom finest (pixel) to coasest: Anneal until convegence: Apply Gibbs sampling using model (2) Update each egion s label based on Gibbs sampling Apply tansition equations (4) and (7) It is difficult to ascetain in theoy whethe this algoithm conveges to the global minimum just like the Gibbs sample does [5]. Ou cuent difficulty lies in the evaluation of the convegence at intemediate levels in the hieachy. Each pai of nodes can convege to one of fou possible configuations shown in Figue 3. The possibilities Figue 3a-c ae acceptable if we ae to each a good minimum point (if not a global minimum) afte the last level in the hieachy conveged. Howeve, should Figue 3d occu then a global minimum will neve be eached. Theefoe, convegence to a good local minimum hinges on a delicate balance of Φ (s), and β (s), at each level s. This balance seems to be achieved in ou peliminay expeiments. Futhemoe, we ae cuently investigating whethe convegence can be poven fomally. Othe moe ceative models can also be used whee the elationships between the egions ae moe complex combinations of mean-based and edge-based calculations. 5 Results and Discussion Results ae pesented on a colo image. The pixel dissimilaity citeion Φ was chosen to be the vecto angle measue following [4] as the image has some intensity diffeences (e.g. shading). Futhemoe, the egion mean is eplaced by its vecto angle analogue, namely, the fist pincipal component of the covaiance matix of the pixel vectos. This value epesents the most pevalent vecto diection in the egion. Results fo the oiginal edge-based model fom [11] ae shown in Figue 4. Model (2) encodes only distances between individual pixels and not fo example distances between egion pototypes [4, 10]. Theefoe, egions connected by a slowly vaying gadient will be meged as can be clealy seen. Howeve, when the mean-based model is used egion spilling does not eadily occu. The esults in in Figue 5 show image segmentation using the mean model. Region spilling is less pevalent than in the classic Potts model case howeve thee is still some A B A B A B A B (a) (b) (c) (d) Figue 3. Thee ae fou possible outcomes when combining two nodes A and B which epesent pixels o egions: (a) nodes that ae supposed to be meged ae meged and the enegy is loweed, (b) nodes that ae supposed to be sepaated ae kept sepaated and the enegy stays the same, (c) nodes that ae supposed to be meged ae not meged and the enegy stays the same, and (d) nodes that ae supposed to be sepaated ae meged and the enegy is inceased. Outcomes (a)-(c) ae desiable in that (a) and (b) lead us close to a global minimum while option (c) hopefully delays the inevitable and will lead to nodes meging at a highe level in the hieachy. Option (d) should not occu as the enegy fomulation does not pemit it unless thee ae too few labels to distibute between the node and its neighbos. occuing. The distance calculation is now based on nonlocal values (i.e., means) and, theefoe, the andom field is no longe exactly model (2). We have pesented hieachical egions based on a egion mean model. In contast to the hieachical egionbased Potts model [11] the new model exhibits less egion spilling due to the smoothing effects of using means to epesent egions instead of edge pixels. Howeve, some egion meging pesists. We ae cuently investigating whethe the meging effects ae caused by the choice of distance metic, a deficiency in the Potts model (o the hieachical Potts model) o both. This modelling flexibility demonstates an inheent benefit to the hieachical method beyond the computational acceleation, namely, that of being able to model egion inteactions though new genealizations at the highe hieachies. Finally, we ae investigating whethe othe models might impove segmentation esults and hieachical Potts can be shown (o not) to convege to a global minimum. Refeences [1] J. Angulo, and J. Sea, Colo segmentation by odeed megings, IEEE ICIP, Vol. 2, pp , Bacelona: Septembe [2] A. Babu and S.C. Zhu, Gaph Patition by Swendsen-Wang Cut, IEEE Tans. on Patten Analy- 4 Poceedings of the 3d Canadian Confeence on Compute and Robot Vision (CRV 06)

5 (a) Oiginal (b) Final segmentation (12 Levels) (a) β =0.020 (b) β =0.040 Figue 4. Colo image segmentation esults with model (2) using β =0.015 (image pixels wee fist nomalized to unit length). Upon close examination (b) shows egion spilling fom the sea to the shit (uppe cental pat of image) and fom the sand to the coat and leg (lowe ight pat). sis and Machine Intelligence, vol. 27, no. 8, pp , [3] Z. Kato, M. Bethod, and J. Zeoubia, A Hieachical Makov Random Field Model and Multitempeatue Annealing fo Paallel Image Classification, Gaphical Models and Image Pocessing, vol. 58, no. 1, 1996, pp [4] P. Fieguth and S. Wesolkowski, Highlight and Shading Invaiant Colo Image Segmentation Using Simulated Annealing, Enegy Minimization Methods in Compute Vision and Patten Recognition III, Sophia- Antipolis, Fance, Septembe 2001, pp Figue 5. Colo image segmentation esults with the mean model using diffeent β values. [9] G. Winkle, Image Analysis, Random Fields and Dynamic Monte Calo Methods, Spinge-Velag, Belin, Gemany, [10] S. C. Zhu and A. Yuille, Region competition: unifying snakes, egion gowing, and Bayes/MDL fo multiband image segmentation, IEEE Tansactions on Patten Analysis and Machine Intelligence, Vol. 18, No. 9, pp , Sept [11] S. Wesolkowski, and P. Fieguth, Hieachical Region-Based Gibbs Random Field Image Segmentation, Intenational Confeence on Image Analysis and Recognition, Poto, Potugal, Sept 2004, pp [5] S. Geman and D. Geman, Stochastic Relaxation, Gibbs Distibutions, and the Bayesian Restoation of Images, IEEE Tans-PAMI, Vol. 6, No. 6, [6] R. M. Haalick and L. G. Shapio, Compute and Robot Vision, Vol. 1, Addison-Welsey, [7] S. Z. Li, Makov Random Field Modelling in Image Analysis, Spinge: Tokyo, Japan, [8] A. Temeau, and N. Boel, A Region Gowing and Meging Algoithm to Colo Segmentation, Patten Recognition, vol. 30, no. 7, pp , Poceedings of the 3d Canadian Confeence on Compute and Robot Vision (CRV 06)