Colour Image Segmentation using Texems

Transcription

1 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS 1 Colour Image Segmentaton usng Texems Xanghua Xe and Majd Mrmehd Department of Computer Scence, Unversty of Brstol, Brstol BS8 1UB, England {xe,majd}@cs.brs.ac.u Abstract We present two methods to perform colour mage segmentaton usng a generatve threedmensonal model that s based on the assumpton that an mage can be generated through an overlapped placement of a few prmtve, exemplar mage patches,.e. texems. Multscale analyss s used n order to capture suffcent mage features and pxel neghbourhood nteractons at relatvely lower computatonal costs. Expermental results on synthetc and real mages are presented to demonstrate ths as a promsng alternatve approach to popular dscrmnatve methods. 1 Introducton Numerous features have been reported for the beneft of mage segmentaton, ncludng hstogram propertes, co-occurrence matrces, local bnary patterns, fractal dmenson, Marov random feld features, multscale multdrectonal flter responses, and smply pxel colour. These features are then spatally and/or spectrally grouped together to form mage regons. Regon growng, merge-splt, Bayesan classfcaton, and neural networ classfcaton are examples of common technques appled to acheve the latter. Most colour mage segmentaton technques are derved from methods desgned for graylevel mages, usually tang one of three forms: (1) Processng each channel ndvdually by drectly applyng graylevel based methods [Caell and Reye, 1993, Handl and Havlce, 2002]: The channels are assumed ndependent and only ther ntra-spatal nteractons are consdered. (2) Decomposng the mage nto lumnance and chromatc channels [Paschos et al., 1999, Dubusson-Jolly and Gupta, 2000]: After transformng the mage data nto the desred (usually applcaton dependent) colour space, texture features are extracted from the lumnance channel whle chromatc features are extracted from the chromatc channels, each n a specfc manner. (3) Combnng spatal nteracton wthn each channel and nteracton between spectral channels [Jan and Healey, 1998, Bennett and Khotanzad, 1998, Palm, 2004, Mrmehd and Petrou, 2000]: Graylevel texture analyss technques are appled n each channel, whle pxel nteractons between dfferent channels are also taen nto account. Also, c The copyrght of ths document resdes wth ts authors. It may be dstrbuted unchanged freely n prnt or electronc forms.

2 2 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS some wors perform global colour clusterng analyss, followed by spatal analyss n each ndvdual stac. The mportance of extractng correlaton between the channels for colour texture analyss has been addressed by several authors. For example, n [Jan and Healey, 1998], Jan and Healey used Gabor flters to obtan texture features n each channel and opponent features that capture the spatal correlaton between the channels. Orgnal 3D models to analyse colour textures have also been developed, where the spatal and spectral nteractons are smultaneously handled, e.g. [Jojc et al., 2003]. The man dffcultes arse n effectvely representng, generalsng, and dscrmnatng three dmensonal data. The eptome [Jojc et al., 2003] provdes a compact 3D representaton of colour textures. The mage s assumed to be a collecton of eptomc prmtves relyng on raw pxel values n mage patches. The neghbourhood pxels of a central pxel n a patch are assumed to be statstcally condtonally ndependent. A hdden mappng gudes the relatonshp between the eptome and the orgnal mage. Ths compact representaton nherently captures the spatal and spectral nteractons smultaneously. The eptome model nspred the development of the texem model [Xe and Mrmehd, 2007], a compact mxture representaton for colour mages, used for novelty based defect detecton. In ths paper, we use a smple generatve model to represent and segment colour mages. The spatal arrangement and nterspectral propertes of pxels are modelled smultaneously wthout decomposng mages nto separate channels. In secton 2, the colour texem model s presented. Secton 3 descrbes two segmentaton methods based on ths model. Texem groupng s dscussed n Secton 4 for multmodal textures. Some expermental results are gven n Secton 5. Secton 6 concludes the paper. 2 Modellng Colour Images Representng colour mages usng 3D space models s consdered a challengng tas n that t s dffcult to eep a compact representaton and stll suffcently characterse the mage data. In [Jojc et al., 2003], Jojc et al. ntroduced the eptome model as a small, condensed representaton of a gven mage whch contans ts prmtve shapes and textural elements. The recently proposed texem (texture exemplar) model [Xe and Mrmehd, 2007] s based on the same assumpton that a gven mage can be generated from a collecton of mage patches and the varaton n placement results n appearance varatons n the mages. However, unle the eptome, the texem model dscards the hdden mappng and uses multple, much smaller eptomc representatons. In [Xe and Mrmehd, 2007], t was shown that texems s a much more effcent way of performng novelty detecton n colour mages. Each of the texems learnt from the mage contan partal degrees of mage mcro-structures. In other words, texems are mplct representatons of mage prmtves, as opposed to textons [Julesz, 1981] whch are explct representatons and very often use base functons [Zhu et al., 2005]. Each texem m s defned by a mean, µ, and a correspondng varance, ω,.e. m = {µ, ω}. An mage s then consdered as a superposton of patches of varous szes. Ths forces the mage propertes not nto a sngle texem, but a famly of them. We use a mxture model to learn the texems whch together characterse a gven mage. The orgnal mage I s broen down nto a set of P overlappng patches Z = {Z } =1 P, each contanng pxels from a subset of mage coordnates. For smplcty, square patches of sze d = N N are used. We assume that there exst K texems, M = {m } =1 K, K P, for mage I such that each patch n Z can

3 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS 3 be generated from a texem wth certan added varatons: p(z θ ) = p(z µ, ω ) = N (Z j, ; µ j,, ω j, ), (1) j S where θ denotes the th texem s parameters wth mean µ and varance ω, N (.) s a Gaussan dstrbuton over Z j,, S s the patch pxel grd, µ j, and ω j, denote mean and varance at the jth pxel poston n the th texem. The mxture model s gven by: p(z Θ) = K p(z θ )α, (2) =1 where Θ = (α 1,..., α K, θ 1,..., θ K ), and α s the pror probablty of th texem constraned by K =1 α = 1. The Expectaton and Maxmsaton (EM) technque can be used to estmate the model parameters. The E-step nvolves a soft-assgnment of each patch Z to texems, M, wth an ntal guess of the true parameters, Θ. We denote the ntermedate teraton t parameters as Θ (t). The probablty that patch Z belongs to the th texem may then be computed usng Bayes rule: p(m Z, Θ (t) ) = The M-step then updates the parameters accordng to: p(z m, Θ (t) )α K =1 p(z m, Θ (t) )α. (3) ˆα = 1 P P p(m Z, Θ (t) ), ˆµ = { ˆµ j, } j S, ˆω = { ˆω j, } j S, =1 ˆµ j, = P =1 Z j, p(m Z, Θ (t) ) =1 P p(m Z, Θ (t), (4) ) ˆω j, = P =1 (Z j, ˆµ j, )(Z j, ˆµ j, ) T p(m Z, Θ (t) ) =1 P p(m Z, Θ (t). ) The E-step and M-step are terated untl the estmatons stablse or the rate of mprovement of the lelhood falls below a pre-specfed convergence threshold. 3 Colour Image Segmentaton Clearly each mage patch from an mage has a measurable relatonshp wth each texem accordng to the posteror, p(m Z, Θ), whch can be convenently obtaned usng Bayes rule n (3). Thus, every texem can be vewed as an ndvdual textural class component, and the posteror can be regarded as the component lelhood wth whch each pxel n the mage can be labelled. Based on ths, we present two dfferent multscale approaches to carry out segmentaton. One performs segmentaton at each level separately, and then updates the label probabltes from coarser to fner levels, and the other smplfes the procedure by learnng the texems across the scales to gan effcency.

4 4 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS 3.1 Segmentaton wth nterscale post-fuson Varous szes of texems are necessary to capture suffcent mage propertes. Alternatvely, the same sze texems can be appled to a multscale mage, as n [Xe and Mrmehd, 2007], where texems were generated from each scale ndependently for novelty detecton. Detecton results from ndvdual scales were then combned to produce a fnal novelty defect map. For mage segmentaton, however, the fuson procedure s more nvolved, e.g. a relaxaton process [Mrmehd and Petrou, 2000] can be used to update the class probabltes from coarser to fner levels. We frst layout the mage n multscale. Besdes computatonal effcency, explotng nformaton at multscale offers other advantages. Charactersng a pxel based on local neghbourhood pxels can be more effectvely acheved by examnng varous neghbourhood relatonshps. A smple Gaussan pyramd was found to be suffcent. Let us denote I (n) as the nth level mage of the pyramd, Z (n) as all the patches extracted from I (n), and l as the total number of levels. We then extract texems from ndvdual pyramd levels. Smlarly, let m (n) denote the nth level of multscale texems and Θ (n) the assocated parameters. Durng the EM process, the stablsed estmaton of a coarser level s used as the ntal estmaton for the fner level,.e. ˆΘ (n,t=0) = Θ (n+1), whch helps speed up the convergence and acheve a more accurate estmaton. For segmentaton, each pxel needs to be assgned a class label, c = {1, 2,..., K}. We can perform ths labellng usng the measurable relatonshp between each patch at ts central pxel poston and the texems, as gven n (3). So, at each scale n, there s a random feld of class labels, C (n). The probablty of a partcular mage patch, Z (n), belongng to a texem (class), c =, m (n) smplfed as p(c (n) Z (n), s determned by the posteror probablty, p(c =, m (n) ), gven by: p(c (n) Z (n) ) = p(z (n) K =1 p(z(n) m (n) )α (n) m (n) )α (n) Z (n), Θ (n) ),, (5) whch s equvalent to the stablsed soluton of (3). The class probablty at gven pxel locaton (x (n), y (n) ) at scale n then can be estmated as p(c (n) (x (n), y (n) )) = p(c (n) Z (n) ). Thus, ths labellng assgnment procedure ntally parttons the mage n each ndvdual scale. As the mage s lad herarchcally, there s nherted relatonshp among parent and chldren pxels. Ther labels should also reflect ths relatonshp. Next, buldng on ths ntal labellng, the parttons across all the scales are fused together to produce the fnal segmentaton map. The class labels c (n) are assumed condtonally ndependent gven the labellng n the coarser scale c (n+1). Thus, each label feld C (n) s assumed only dependent on the prevous coarser scale label feld C (n+1). Ths offers effcent computatonal processng, whle preservng the complex spatal dependences n the segmentaton. The label feld C (n) becomes a Marov chan structure n the scale varable n: p(c (n) c (>n) ) = p(c (n) c (n+1) ), (6) where c (>n) = {c () } =n+1 l are the class labels at all coarser scales greater than the nth, and p(c (l) c (l+1) ) = p(c (l) ) as l s the coarsest scale. The coarsest scale segmentaton s drectly based on the ntal labellng.

5 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS 5 A quadtree structure for the multscale label felds s assumed, and c (l) only contans a sngle pxel, although a more sophstcated context model can be used to acheve better nteracton between chld and parent nodes, e.g. a pyramd graph model [Cheng and Bouman, 2001]. The transton probablty p(c (n) c (n+1) ) can be effcently calculated numercally usng a looup table. The label assgnment at each scale are then updated, from coarsest to the fnest level, accordng to the jont probablty of the data probablty and the transton probablty: { ĉ(l) = arg max c (l) log p(c (l) (x (l), y (l) )), ĉ (n) = arg max c (n){log p(c (n) (x (n), y (n) )) + log p(c (n) c (n+1) )} n < l. (7) The segmented regons wll be smooth and small solated holes are flled. 3.2 Segmentaton usng branch parttonng Startng from the pyramd layout descrbed n Secton 3.1, each pxel n the fnest level can trace ts parent pxel bac to the coarsest level formng a unque route or branch. In Secton 3.1, the condtonal ndependence assumpton amongst pxels wthn the local neghbourhood maes the parameter estmaton tractable. Here, we assume pxels n the same branch are condtonally ndependent,.e. p(z θ ) = p(z µ, ω ) = n l N (Z (n) ; µ (n), ω (n) ), (8) where Z here s a branch of pxels, Z (n), µ (n), and ω (n) are the colour pxel at level n n th branch, mean at level n of th texem, and varance at level n of th texem, respectvely. Ths s essentally the same form as (1), hence, we can stll use the EM procedure descrbed prevously to derve the texem parameters. However, the mage s not parttoned nto patches, but rather lad out n multscale frst and then separated nto branches. The pxels are collected across scales, nstead of from ts neghbours. The class labels are then drectly gven by the component lelhood, agan usng Bayes rule, p(c Z ) = p(m Z, Θ). Thus, we smplfy the approach presented n Secton 3.1 by avodng the nter-scale fuson after labellng each scale. 4 Texem Groupng for Multmodal Texture A textural regon may contan multple vsual elements and dsplay complex patterns. A sngle texem mght not be able to fully represent such textural regons, hence, several texems can be grouped together to jontly represent multmodal texture regons. Here, we use a smple but effectve method proposed by Manduch [Manduch, 1999, 2000] to group texems. The basc strategy s to group some of the texems based on ther spatal coherence. The groupng process smply taes the form: ˆp(Z c) = 1ˆβ c G c p(z m )α, ˆβ c = G c α, (9) where G c s the group of texems that are combned together to form a new cluster c whch labels the dfferent texture classes, and ˆβ c s the pror for new cluster c. The mxture model

6 6 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS can thus be reformulated as: p(z Θ) = ˆK ˆp(Z m ) ˆβ c, (10) c=1 where ˆK s the desred number of texture regons. Equaton (10) shows that pxel n the centre of patch Z wll be assgned to the texture class c whch maxmses ˆp(Z c) ˆβ c : c = arg max c ˆp(Z c) ˆβ c = arg max c G c p(z m )α. (11) The groupng n (10) s carred out based on the assumpton that the posteror probabltes of grouped texems are typcally spatally correlated. The process should mnmse the decrease of model descrptveness, D, whch s defned as [Manduch, 1999, 2000]: D = K D j, D j = j=1 p(z m j )p(m j Z )dz = E[p(m j Z ) 2 ] α j, (12) where E[.] s the expectaton computed wth respect to p(z ). In other words, the compacted model should retan as much descrptveness as possble. Ths s nown as the Maxmum Descrpton Crteron (MDC). The descrptveness decreases drastcally when well separated texem components are grouped together, but decreases very slowly when spatally correlated texem component dstrbutons merge together. Thus, the texem groupng should search for smallest change n descrptveness, D. It can be carred out by greedly groupng two texem components, m a and m b, at a tme wth mnmum D ab : D ab = α bd a + α a D b α a + α b 2E[p(m a Z )p(m b Z )] α a + α b. (13) We can see that the frst term n (13) s the maxmum possble descrptveness loss when groupng two texems, and the second term n (13) s the normalsed cross correlaton between the two texem component dstrbutons. Snce one texture regon may contan dfferent texem components that are sgnfcantly dfferent to each other, t s benefcal to smooth the posteror as proposed n [Manduch, 2000] such that a pxel that orgnally has hgh probablty to just one texem component wll be softly assgned to a number of components that belong to the same multmodal texture. After groupng, the fnal segmentaton map s obtaned accordng to (11). 5 Expermental Results Here, we present some expermental results and a bref comparson wth the well-nown JSEG technque [Deng and Manjunath, 2001]. Fg. 1 shows example results on four dfferent texture collages wth the orgnal mage n the frst row, groundtruth segmentatons n the second row, the JSEG result n the thrd row, the proposed nterscale post-fuson method n the fourth row, and the proposed branch partton method n the fnal row. The two proposed schemes have smlar performance, whle JSEG tends to over-segment whch partally arses due to the lac of pror nowledge of number of texture regons.

7 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS 7 Fgure 1: Testng on synthetc mages - frst row: orgnal mage collages, second row: groundtruth segmentatons, thrd row: JSEG results, fourth row: results of the proposed method usng nterscale post-fuson, last row: results of the proposed method usng branch parttonng. Two real mage examples are gven n Fg. 2. For each mage, we show the orgnal mages, ts JSEG segmentaton and the results of the two proposed segmentaton methods. The nterscale post-fuson method produced fner borders but s a slower technque. Fg. 3 focuses on the nterscale post-fuson technque followed by texem groupng. The orgnal mage and the fnal segmentaton are shown at the top. The second row shows the ntal labellng of 5 texem classes for each pyramd level. The texems are grouped to 3 classes as seen n the thrd row. Interscale fuson s then performed and shown n the last row. Note there s no fuson n the fourth (coarsest) scale. In Fg. 4, JSEG agan over-segmented the mages when the texture regons were multmodal n nature. The branch partton method followed by texem groupng segmented the

8 8 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS Fgure 2: Testng on real mages - frst column: orgnal mages, second column: JSEG results, thrd column: results of the proposed method usng nterscale post-fuson, fourth column: results of the proposed method usng branch parttonng. Fgure 3: An example of the nterscale post-fuson method followed by texem groupng - frst row: orgnal mage and ts segmentaton result usng the proposed method wth nterscale post-fuson, second row: ntal labellng of 5 texem classes for each scale, thrd row: updated labellng after groupng 5 texems to 3, fourth row: results of nterscale fuson. mages nto a more plausble number of texture regons. The results shown demonstrate that the two proposed methods are more able n modellng textural varatons than JSEG and are less prone to over-segmentaton. However, t s noted that JSEG does not requre the number of regons as pror nowledge. On the other hand, texem based segmentaton provdes a useful descrpton for each regon and a measurable relatonshp between them. The number of texture regons may be automatcally de-

9 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS 9 Fgure 4: Comparson - from left on each row: orgnal mage, JSEG result, the branch partton method followed by texem groupng. termned usng model-order selecton methods, such as MDL. The post-fuson and branch partton schemes acheved comparable results, whle the branch partton method s faster. However, a more thorough comparson s necessary to draw complete conclusons, whch s part of our future wor. 6 Conclusons We presented two colour mage segmentaton methods based on the texem model. We also showed how to group texems as a potentally useful tool for manpulatng them. Future wor wll focus on methods to automatcally estmate the number of texture regons and to further speed-up the texem learnng process. References J. Bennett and A. Khotanzad. Multspectral random feld models for synthess and analyss of color mages. IEEE Transactons on Pattern Analyss and Machne Intellgence, 20(3): , T. Caell and D. Reye. On the classfcaton of mage regons by colour, texture and shape. Pattern Recognton, 26(4): , H. Cheng and C. Bouman. Multscale Bayesan segmentaton usng a tranable context model. IEEE Transactons on Image Processng, 10(4): , Y. Deng and B. Manjunath. Unsupervsed segmentaton of color-texture regons n mages and vdeo. IEEE Transactons on Pattern Analyss and Machne Intellgence, 23(8): , M. Dubusson-Jolly and A. Gupta. Color and texture fuson: Applcaton to aeral mage segmentaton and GIS updatng. Image and Vson Computng, 18: , 2000.

10 10 XIE AND MIRMEHDI: COLOUR IMAGE SEGMENTATION USING TEXEMS M. Handl and V. Havlce. A smple multspectral multresoluton Marov texture model. In Internatonal Worshop on Texture Analyss and Synthess, pages 63 66, A. Jan and G. Healey. A multscale representaton ncludng opponent color features for texture recognton. IEEE Transactons on Image Processng, 7(1): , N. Jojc, B. Frey, and A. Kannan. Eptomc analyss of appearance and shape. In IEEE Internatonal Conference on Computer Vson, pages 34 42, B. Julesz. Textons, the element of texture percepton and ther nteractons. Nature, 290:91 97, R. Manduch. Bayesan fuson of color and texture segmentatons. In IEEE Internatonal Conference on Computer Vson, pages , R. Manduch. Mxture models and the segmentaton of multmodal textures. In IEEE Conference on Computer Vson and Pattern Recognton, pages , M. Mrmehd and M. Petrou. Segmentaton of color textures. IEEE Transactons on Pattern Analyss and Machne Intellgence, 22(2): , C. Palm. Color texture classfcaton by ntegratve co-occurrence matrces. Pattern Recognton, 37(5): , G. Paschos, P. Kmon, and P. Valavans. A color texture based montorng system for automated survellance. IEEE Transactons on Systems, Man, and Cybernetcs, 29(1): , X. Xe and M. Mrmehd. TEXEMS: Texture exemplars for defect detecton on random textured surfaces. IEEE Transactons on Pattern Analyss and Machne Intellgence, Accepted. S. Zhu, C. Guo, Y. Wang, and Z. Xu. What are textons? Internatonal Journal of Computer Vson, 62(1-2): , 2005.