Color Image Segmentaton Usng Multspectral Random Feld Texture Model & Color Content Features Orlando J. Hernandez E-mal: hernande@tcnj.edu Department Electrcal & Computer Engneerng, The College of New Jersey Ewng, New Jersey 08628-0718, USA and Alreza Khotanzad E-mal: kha@engr.smu.edu Department of Electrcal Engneerng, Southern Methodst Unversty Dallas, Texas 75275-0338, USA ABSTRACT Ths paper descrbes a color texture-based mage segmentaton system. The color texture nformaton s obtaned va modelng wth the Multspectral Smultaneous Auto Regressve (MSAR) random feld model. The general color content characterzed by ratos of sample color means s also used. The mage s segmented nto regons of unform color texture usng an unsupervsed hstogram clusterng approach that utlzes the combnaton of MSAR and color features. The performance of the system s tested on two databases contanng synthetc mosacs of natural textures and natural scenes, respectvely. Keywords: Color Texture, Multspectral Random Feld Models, Color Texture Segmentaton 1. INTRODUCTION Ths paper descrbes a color texture-based segmentaton system. The approach nvolves characterzng color texture usng features derved from a class of multspectral random feld models and color space. These features are then used n an unsupervsed hstogram clusterng-based segmentaton algorthm to fnd regons of unform texture n the query mage. Texture nformaton has been used n the past for the characterzaton of magery, however the prevously proposed approaches have ether consdered only gray level textures or pxels based color content and not color texture [1]. The utlzed segmentaton algorthm n some approaches has also not been completely unsupervsed [2]. The contrbutons of ths paper are as follows: Texture characterzaton by a mx of multspectral random feld based features and color content features. Development of a completely unsupervsed colortexture-based segmentaton algorthm and demonstraton of ts effectveness. Demonstraton of the effectveness of the proposed approach on two databases contanng 1) synthetc mosacs of natural textures and 2) natural scenes. Related Studes The man aspect to ths work s segmentaton of color texture mages. Accordngly, some salent prevous studes related to ths topc are revewed n ths secton. The texture segmentaton algorthm n [3] consders features extracted wth a 2-D movng average (MA) approach. The 2-D MA model represents a texture as an output of a 2-D fnte mpulse response (FIR) flter wth smple nput process. The 2-D MA model s used for modelng both sotropc and ansotropc textures. The maxmum lkelhood (ML) estmator of the 2-D MA model s used as texture features. Supervsed and unsupervsed texture segmentaton are consdered. The texture features extracted by the 2-D MA modelng approach from sldng wndows are classfed wth a neural network for supervsed segmentaton, and are clustered by a fuzzy clusterng algorthm for unsupervsed texture segmentaton. Mrmehd and Petrou n [4] present an approach to perceptual segmentaton of color mage textures. Intal segmentaton s acheved by applyng a clusterng algorthm to the mage at the coarsest level of smoothng. The mage pxels representng the core clusters are used to form 3D color hstograms that are then used for probablstc assgnment of all other pxels to the core clusters to form larger clusters and categorze the rest of the mage. The process of settng up color hstograms and probablstc reassgnment of the pxels to the clusters s then propagated through fner levels of smoothng untl a full segmentaton s acheved at the hghest level of resoluton. Deng and Manjunath [1] also present a new method (JSEG) for unsupervsed segmentaton of color-texture regons n mages. It conssts of two ndependent steps: color quantzaton and spatal segmentaton. In the frst step, colors n the mage are quantzed to several representatve classes that can be used to dfferentate regons n the mage. Applyng the crteron to local wndows n the class-map results n the J-mage, n whch hgh and low values corresponded to possble boundares and nterors of color-texture regons. A regon growng method s then used to segment the mages based on the multscale J-mages. Ths method s stll pxel based, and we contrast results of mage segmentaton of ths algorthm wth our approach later n ths paper. 2. COLOR TEXTURE CHARACTERIZATION WITH MULTISPECTRAL SIMULTANEOUS AUTOREGRESSIVE MODEL In ths work, the texture of the color mages s characterzed usng a class of multspectral random feld mage model called the Multspectral Smultaneous Autoregressve (MSAR) model [5], [6]. The MSAR model has been shown to be effectve for color texture synthess and classfcaton [5], [6]. For mathematcal smplcty, the model s formulated usng a torodal lattce assumpton. A locaton wthn a two-dmensonal M x M lattce s denoted by s (, j), wth, j beng ntegers from the set J {0, 1,, M-1}. The set of all lattce locatons s defned as Ω {s (, j) :, j J}. The value of an mage observaton at locaton s s denoted by the vector value y(s), and the mage observatons are assumed to have zero mean. The MSAR model relates each lattce poston to ts 141
neghborng pxels, both wthn and between mage planes, accordng to the followng model equaton: P y θ ( r) y ( s r) + ρ w, 1K P j j 1 r Nj y (s) Pxel value at locaton s of the th plane s and r two dmensonal lattces P number of mage planes (for color mages, P 3, representng: Red, Green, and Blue planes) N j neghbor set relatng pxels n plane to neghbors n plane j (only nterplane neghbor sets,.e. N j, j, may nclude the (0,0) neghbor) θ j coeffcents whch defne the dependence of y (s) on the pxels n ts neghbor set N j ρ nose varance of mage plane w (s)..d. random varables wth zero mean and unt varance denotes modulo M addton n each ndex The parameters assocated wth the MSAR model are θ and ρ vectors whch collectvely characterze the spatal nteracton between neghborng pxels wthn and between color planes. These vectors are taken as the feature set f T representng the underlyng color texture of the mage. A least squares (LS) estmate of the MSAR model parameters s obtaned by equatng the observed pxel values of an mage to the expected value of the model equatons [5]. Ths leads to the followng estmates: θˆ and 1 T q q q y s Ω s Ω ρˆ q y 1 T ( y θˆ q ( s ) 2 ) 2 M s Ω θ T T T [ θ θ L θ ] T 1 2 P T T T [ y y L y ( ] T 1 2 P s) j { y ( s r) : N } col r j Ths model has been shown to be effectve for color texture synthess as well n [6], and the resultng synthetc mages can be vsually compared to the orgnal mages, and observed to be very smlar n appearance to the mages from whch the models were derved. j (1) (2) (3) (4) (5) (6) 3. COLOR CONTENT CHARACTERIZATION In addton to modelng color texture, the general color content of the mage s also mportant. Addtonal features focusng on the color alone are also consdered. Ths s done usng the sample mean of the pxel values n the red, green, and blue (RGB) planes. The defned feature vector s: ) ) µ r µ f C ), ) µ g µ wth µˆ s beng the sample mean of the respectve color component. The reason for usng these specfc ratos, nstead of some of the other ratos formed by the other combnatons of the color means, s that ths same relatonshp was used to form ratos of the ρ parameters of the MSAR model used for texture classfcaton n [5] wth very postve results. Also, the reason for usng rato of color means nstead of color means themselves s that such a rato s llumnaton nvarant. Assumng that the observed value at each pxel s a product of llumnaton and spectral reflectance, the ratos of the color means are nvarant to unform changes n llumnaton ntensty (.e. the power of the llumnaton source changes unformly across the spectrum). Ths knd of unform change would cause each µˆ to change by the same scale factor makng the defned ratos nvarant to llumnaton changes. Ths property makes the color-content features more robust. In the event that the denomnator of any of the ratos of the color means goes to zero, the color mean wth a value of zero s changed to a value of one to avod the mathematcal excepton of dvdng by zero. Ths case; however, s very unlkely, snce we are dealng wth textures and natural mages that do not tend to have large areas (.e. the feature extracton sldng wndow to be descrbed later) wth a sold color extreme. The combnaton of f C and f T features s used to represent a color texture regon n ths work. These features are collectvely referred to as Color Content, Color Texture (C 3 T) features. 4. UNSUPERVISED IMAGE SEGMENTATION WITH A HISTOGRAM-BASED CLUSTERING ALGORITHM The segmentaton algorthm used n ths work reles on scannng the mage wth a sldng wndow and extractng C 3 T features from each wndow. These features are then clustered usng an unsupervsed hstogram-based algorthm. Mappng the dentfed clusters back nto the mage doman results n the desred segmentaton. Feature Extracton wth a Sldng Wndow The wndowng operaton conssts of sldng a wndow from left to rght and top to bottom across the mage as llustrated n Fg. 1. M s the sze of the mage n pxels, W s the sze of the wndow n pxels, and D s the sze of the sldng step n pxels. After extensve expermentaton, where the value of D s vared from 1 pxel to W pxels, D s set to 4 pxels for ths work, as ths value yelded the best results. Wth D havng a value of W pxels, the sldng wndows are non-overlappng and adjacent to each other. To fnd the optmum wndow sze for each case, the sze of the wndow W vares from 4 to 28 pxels n ncrements r b (7) 142
of 4 pxels. The best W s found automatcally as descrbed n later sectons. M W W D M Fg. 1. Feature vector extracton wth a sldng wndow As the wndow sze s ncreased, the overlap between areas covered by adjacent wndows ncreases, snce D s constant, and thus the redundancy of nformaton from feature vectors obtaned from larger adjacent wndows ncreases as W ncreases. It was decded not to ncrease D for the larger W values, and thus not to reduce the spatally adjacent vectors redundancy, because as W ncreases the lkelhood of capturng a more heterogeneous mage area ncreases as well. Leavng redundancy between adjacent vectors helps the coheson and convergence of the clusterng process, from a spatal perspectve. The texture bounded by each wndow s characterzed usng the C 3 T features. The neghborhood used for the MSAR model s a set that contans neghbors above, below, to the left, and to the rght of the pxel as llustrated n Fg. 2. The same neghbor set s used for both nter and ntra-planes of the model. X X O X X Fg. 2. Neghbor set used wth the MSAR model Ths neghbor set results n a 20-dmensonal f T. Therefore together wth the two-dmensonal color content feature set, a 22-dmensonal C 3 T feature vector, f, s used to characterze each wndow. Clusterng Algorthm Once all 22-dmensonal f features are extracted from the sldng wndow, they are clustered n the feature space usng an unsupervsed hstogram-based peak clmbng algorthm [7], [8]. The 22-dmensonal hstogram s generated by quantzng each dmenson accordng to the followng: CS d k ( k) f max f INT ( k) f ( k) Q mn ( k) f mn ( k) CS( k) + 1 ; ; k 1, 2, K, N (8) k 1, 2, K, N N total number of features (22 n ths case) CS(k) length of the kth sde of hstogram cell f max (k) maxmum value of the kth C 3 T features f mn (k) mnmum value of the kth C 3 T features Q total number of quantzaton levels d k kth ndex for a hstogram cell Snce the dynamc range of the vectors n each dmenson can be qute dfferent, the cell sze for each dmenson would be dfferent. Hence the cells wll be hyperboxes. Next, the number of feature vectors fallng n each hyperbox s counted and ths count s assocated wth the respectve hyperbox creatng the requred hstogram. After the hstogram s generated n the feature space, a peak clmbng clusterng approach s utlzed to group the features nto dstnct clusters. Ths s done by locatng the peaks of the hstogram. In Fg. 3 ths peak clmbng approach s llustrated for a two-dmensonal space example. 8 7 6 5 4 3 2 1 40 28 12 15 50 30 5 1 10 25 4 2 1 4 4 1 20 8 3 1 15 7 5 5 2 35 45 20 8 18 80 2 3 10 5 1 2 3 4 5 6 7 8 9 Fg. 3. Illustraton of the Peak Clmbng approach for a two-dmensonal feature space example The number n each cell (hyperbox) represents a hypothetcal count for the feature vectors captured by that cell. By examnng the counts of the 8-neghbors of a partcular cell, a lnk s establshed between that cell and the closest cell havng the largest count n the neghborhood. At the end of the lnk assgnment, each cell s lnked to one parent cell, but can be parent of more than one cell. A peak s defned as beng a cell wth the largest densty n the neghborhood,.e. a cell wth no parent. A peak and all the cells that are lnked to t are taken as a dstnct cluster representng a mode n the hstogram. Once the clusters are found, the wndows assocated wth (9) 143
features grouped n the same cluster are tagged as belongng to the same category. A major component of ths algorthm s the number of quantzaton levels assocated wth each dmenson. To decde ths parameter, the total number of non-empty cells and the percentage of them capturng only one vector for each selecton of quantzaton levels are examned. The best number of quantzaton levels s selected as the largest one that maxmzes the measure below [16]. ( N c N u ) N u M N c number of non-empty cells N u number of cells capturng only one sample (10) The algorthm also ncludes a spatal doman cluster valdaton step. Ths step nvolves constructng a matrx B for each cluster m as: B B m m (, j) 1 (, j) 0 f sample cluster m otherwse (11) The (, j) ndex corresponds to the locaton of a sldng wndow. A cluster s consdered compact f only a very small number of ts 1-elements have a 0-element neghbor,.e. a cluster s consdered vald (compact) f only a very small number of ts elements have neghborng elements that do not belong to that cluster. A cluster that does not pass ths test s merged wth a vald cluster that has the closest centrod to t. Durng the segmentaton process, the best wndow sze for scannng the mage s chosen n an unsupervsed fashon. The optmum wndow sze s obtaned by sweepng the mage wth varyng wndow szes (4 to 28 pxels n steps of 4 pxels), and choosng the smallest one out of at least two consecutve wndow szes that produce the same number of clusters. 5. RESULTS The performance of the system on two dfferent databases s reported n ths secton. Test Databases Two databases are used to test the performance of the proposed approach. These databases are referred to as Natural Texture Mosacs Database, and Natural Scenes Database. Natural Textures Mosacs Database: The Natural Texture Mosacs Database s a collecton of mages that are mosacs of natural textures constructed from texture mages avalable n [9]. Ths collecton ncludes 200 128 x 128 mages wth mosacs put together at random n terms of the arrangement and the type of the texture regons used. Ths database s constructed to measure and nvestgate the effectveness of the algorthm n a controlled envronment. Natural Scenes Database: The Natural Scenes Database s a collecton of mages, whch s comprsed of natural scenes avalable n [10]. Ths collecton has 400 mages of varous szes that nclude 120 x 80, 80 x 120, 128 x 85, and 85 x 128 pxels. Ths database was constructed to test the performance on real magery. Segmentaton Results The performance of the proposed segmentaton algorthm and the assocated features s llustrated n Fgs. 4 and 5. Fg. 4 shows fve mages each contanng a number of dfferent textures. These mage mosacs are created from texture samples avalable n [9]. Below each mage the segmentaton result s presented n the form of a gray-level mage wth pxels belongng to the same texture havng the same gray level. In the next row, the boundares of the segmented regons are shown as supermposed whte lnes. At the top of the fgures, the sze of the optmal wndow found by the algorthm s shown. Fg. 5 shows the segmentaton results for several natural scene mages. These natural scene mages are avalable n [10]. It s observed that the proposed algorthm performs qute well and s capable of localzng unform color textures n each mage. In Fg. 4 and Fg. 5, we also compare the results of our approach wth the mage segmentaton results acheved usng the JSEG method descrbed n [1]. The JSEG results were obtaned from applyng the mages to the programs made avalable by the JSEG authors at the Internet ste http://maya.ece.ucsb.edu/jseg/. The obtaned regon boundares are supermposed on the orgnal mages. The JSEG results are dsplayed n the last rows of Fgs. 4 and 5. It can be seen that our segmentaton results have a better match wth perceptual boundares n the mages. The JSEG method over segments most of the natural scene mages and msses or mslabels some boundares n mosac mages. However; the approach proposed n ths work s 3 to 5 tmes more computatonally ntensve than the JSEG method, e.g. t takes about 15 seconds to segment a 128 x 128 pxels mage wth ths method on a Pentum II 400MHz processor versus about 5 seconds wth the JSEG method. 6. CONCLUSIONS In ths work, a novel color texture-based approach to mage segmentaton s developed. Features derved from the Multspectral Autoregressve (MSAR) random feld model wth a 4-neghbor set, and the RGB color space represented by the ratos of the true color plane means are used to characterze the color texture content of the mage. These features are extracted from the mage usng a samplng wndow that sldes over the entre mage, and are used n conjuncton wth an unsupervsed clusterng-based segmentaton algorthm to segment the mages nto regons of unform color texture. The mage regons are obtaned by mappng back to the spatal doman of the mage the sgnfcant clusters obtaned n the 22-dmensonal feature space durng the clusterng process. The effectveness of the approach has been demonstrated usng two dfferent databases contanng synthetc mosacs of natural textures and natural scenes. Furthermore, applcatons of ths new perceptually compatble mage segmentaton method are possble n the areas of vdeo processng and event detecton, and vdeo database and retreval systems. 7. REFERENCES [1] Y. Deng and B. S. Manjunath, Unsupervsed Segmentaton of Color-Texture Regons n Images and Vdeo, IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 23, no. 8, 2001, pp. 800-810. 144
[2] J. R. Smth and S-F. Chang, Integrated Spatal and Feature Image Query, Multmeda Systems ACM - Sprnger-Verlag 1999, vol. 7, no. 2, 1999, pp. 129-140. [3] K. B. Eom, Segmentaton of monochrome and color textures usng movng average modelng approach, Elsever Scence B. V. Image and Vson Computng, no. 17, 1999, pp. 233-244. [4] M. Mrmehd and M. Petrou, Segmentaton of Color Textures, IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 22, no. 2, 2000, pp. 142-159. [5] J. W. Bennett, Modelng and Analyss of Gray Tone, Color, and Multspectral Texture Images by Random Feld Models and Ther Generalzatons, Ph.D. Dssertaton, Southern Methodst Unversty, 1997. [6] J. W. Bennett and A. Khotanzad, Multspectral Random Feld Models for Synthess and Analyss of Color Images, IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 20, no. 3, 1998, pp. 327-332. [7] A. Khotanzad and J. Y. Chen, Unsupervsed Segmentaton of Textured Images by Edge Detecton n Multdmensonal Features, IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 11, no. 4, 1989, pp. 414-421. [8] A. Khotanzad and A. Bouarfa, Image Segmentaton by a Parallel, Non-Parametrc Hstogram Based Clusterng Algorthm, Pattern Recognton, vol. 23, no. 9, 1990, pp. 961-963. [9] Vson and Modelng Group, MIT Meda Laboratory, Vson Texture (VsTex) database, http://wwwwhte.meda.mt.edu/vsmod/, 1995. [10] Corel Corporaton, Professonal Photos CD-ROM Sampler SERIES 200000, 1994. W 12 W 24 W 08 W 08 W 12 Fg. 4. Segmentaton results for four natural texture mosac mages, 1 st row: Orgnal mage, 2 nd row: Segmentaton results, 3 rd row: Texture boundares correspondng to segmentaton results, 4 th row: Segmentaton usng JSEG method 145
W08 W12 W08 W04 W04 W08 W08 W12 W04 W04 Fg. 5. Segmentaton results for eght natural scene mages, 1 st row: Orgnal mage, 2 nd row: Segmentaton results, 3 rd row: Texture boundares correspondng to segmentaton results, 4 th row: Segmentaton usng JSEG method 146