Probabilistic Classification of Image Regions using an Observation-Constrained Generative Approach

Transcription

1 9 Probabilisti Classifiation of mage Regions using an Observation-Constrained Generative Approah Sanjiv Kumar, Alexander C. Loui 2, and Martial Hebert The Robotis nstitute, Carnegie Mellon University, Pittsburgh, PA, USA 2 maging Siene and Tehnology Lab, Eastman Kodak Company, Rohester, NY, USA {skumar, hebert}@ri.mu.edu, alexander.loui@kodak.om Abstrat n generi image understanding appliations, one of the goals is to interpret the semanti ontext of the sene (e.g., beah, offie et.). n this paper, we propose a probabilisti region lassifiation sheme for natural sene images as a priming step for the problem of ontext interpretation. n onventional generative methods, a generative model is learnt for eah lass using all the available training data belonging to that lass. However, if a set of newly observed data has been generated beause of the subset of the model support, using the full model to assign generative probabilities an produe serious artifats in the probability assignments. This problem arises mainly when the different lasses have multimodal distributions with onsiderable overlap in the feature spae. We propose an approah to onstrain the lass generative probability of a set of newly observed data by exploiting the distribution of the new data itself and using linear weighted mixing. A KL-Divergene-based fast model seletion proedure is also proposed for learning mixture models in a sparse feature spae. The preliminary results on the natural sene images support the effetiveness of the proposed approah. Keywords mage region lassifiation, generative model, semanti interpretation, image segmentation.. NTRODUCTON AUTOMATC extration of the semanti ontext of a sene is useful for image indexing and retrieval, roboti navigation, surveillane, robust objet detetion and reognition, and auto-albuming et. Reent literature reveals inreasing attention to this field []-[3]. However, most of the attempts have been made to extrat the ontext of a sene at a high level in the abstration hierarhy. For example, Torralba et al. [] represent the ontext by using the power spetra at different spatial frequenies while Vailaya [2] uses the edge oherene histograms to differentiate between natural and urban senes. Limited attention has been paid to the task of speifi ontext generation from a sene (sene lassifiation), e.g., if a sene is a beah or an offie. The main hurdle in suh ontext generation is that it requires not only the knowledge of the regions or objets in the image, but the semanti information ontained in their spatial arrangement as well. The motivation for our work omes from the following paradox of sene lassifiation. n absene of any a-priori information, the sene lassifiation task requires the knowledge of regions and objets ontained in the image. On the ontrary, it is inreasingly being reognized in vision ommunity that ontext information is neessary for reliable extration of the image regions and objets [], [3]. To solve this paradox, an iterative feedbak sheme an be envisaged, whih refines the sene ontext and image region hypotheses iteratively. Under this paradigm, an obvious hoie to the region lassifiation sheme is one that allows easy modifiation of the initial lassifiation without requiring to lassify the regions afresh. Probabilisti lassifiation of image regions an provide great flexibility in future refinement using the Bayesian approah as the ontext information an be enoded as improved priors. n this paper, we deal with the probabilisti lassifiation of the image regions belonging to the senes primarily ontaining natural objets, e.g., water, sky, sand, skin, et. as a priming step for the problem of sene ontext generation. Several tehniques have been proposed to lassify various image regions in distint ategories, e.g., [4], [5]. However, these are primarily based on disriminative approahes leading to hard lass assignments and hene are not suitable for the iterative refinement sheme mentioned above. n the onventional informative shemes, a generative model for eah lass is learnt using all the available training data belonging to that lass and newly observed data is assigned a probability based on the learnt model. However, it is possible that a set of newly observed data may have been generated beause of the subset of the full generative model support, and using the full model to assign generative probabilities an produe serious artifats in the probability assignments. For example, in the training set, various pixels assoiated with the lass tree may have wide variations in the olor and texture depending on the season, illumination onditions, and the sale. A generative model for lass tree will try to apture all these variations in the same model. Hene, while assigning the generative probabilities to the newly observed data, the learnt generative model may assign high probability to not only the pixels belonging to the lass tree, but also to those that may have some semblane to tree

2 92 data (in some arbitrary ombination of illumination and other physial onditions). Similar analogy holds for the data belonging to other lasses also. This problem arises mainly when different lasses have multimodal distributions that are lose in the feature spae. Another problem with the generative models is that they tend to give more weight to the regions in the feature spae that ontain more data, instead of emphasizing on the disriminative boundary between the data belonging to different lasses. This implies that the data near the boundary will be assigned almost similar probabilities irrespetive of their lass affiliations. We propose to alleviate these problems by using the simple observation that the newly observed data is usually generated by a small support of the overall generative model. n our previous tree example, this means that the data belonging to the tree lass in a new test image is usually generated during the same season as well as under relatively homogeneous illumination and other physial onditions. Thus, the distribution of the newly observed data an be used to onstrain the overall generative model while produing the generative lass density maps for that data. That is why the proposed tehnique has been named as the observationonstrained generative approah.. GENERATVE APPROACH n the first stage of our approah, we use two well-known tehniques, i.e., supervised learning using the labeled training data, and the unsupervised learning using the newly observed test data. The main ontribution of our approah lies in the next stage where the outputs of these two tehniques are merged using a Bayesian sheme. n the beginning of the following disussion, we briefly review the unsupervised and supervised learning applied in the present ontext. Let X be the set of training data assoiated with lass, where d n X = { xk : x k R } k = and C be the lasses of interest. The set ontaining all the C lasses is referred to as the reognition voabulary. The dimension of the feature spae, d is the number of bands in the multiband input images. Eah band represents a olor or a texture feature assoiated with the image pixels. To apture this inherent multimodality of the data, we assume a mixture of Gaussian generative model for the data. Mixture models an approximate any ontinuous density to arbitrary auray provided the model has suffiiently large number of omponents and the parameters of the model are hosen orretly [6]. The lass onditional density funtion of a data point x x k X is given by, M k ) = xk m, ) P( m ) m= where M is the number of omponents in the mixture model. () n this model, eah data point belonging to lass is generated by first hoosing a Gaussian omponent with probability P(m ) and then generating the data point with probability k x x k m, ) = 2πC x k m, ), whih is a Gaussian given by, exp ( x / 2 2 m µ m m k T m µ ) C m ( x k µ where is the mean and C is the ovariane matrix of the omponent m, belonging to lass. The parameters θ of the generative model in (), i.e., µ m m, C and P(m ) for eah omponent m are learnt using the standard Maximum Likelihood formulation using the Expetation Maximization (EM) optimization tehnique [7]. Thus, ˆ θ = arg max θ n M k = m= xk m, ) P( m ) n the above formulation, the data points in the set X have been assumed to be onditionally independent given θ. For a given test image, the newly observed dataset is defined as m ) (2) (3) where N is the number of pxiels in. The probability of assoiation of eah with a given lass in the reognition voabulary (given by ()) yields a Class Density Map over the image. Now, the aim is to onstrain the probabilities ontained in these maps by enforing the statistis of the newly observed data obtained from the test image. To do this, at first, the newly observed data is soft lustered in different groups in an unsupervised manner. For soft lustering, we again use the mixture of the Gaussian model. Aording to this, the probability of a newly observed data point, d N X = { xt : xt R } t= x t x is given as, K t ) = xt j ) P( j ) j= where j = K are the lusters and x j ) ~ Ν (,C ) x t t µ j similar to (2). However, instead of traditional lustering (where eah image pixel is assigned to a partiular luster), we utilize the probability of assoiation of eah pixel with a t luster j, i.e., P( j x ). This leads to soft or probabilisti lustering and the maps representing these probabilities for eah luster are alled Cluster Maps. These maps enable the refinement of the Class Density Maps by onstraining the overall generative model. To estimate the parameters of the unsupervised learning model mentioned in (4), similar EM formulation is used as in (3) exept that m and x k are replaed by j and x t (4) respetively along with the j

3 93 suitable ardinalities of their orresponding sets. Given the test image, the first step towards onstraining the Class Density Maps using soft lustering involves finding the lass that has highest probability of being represented by a Cluster Map. This is done by maximizing the onditional probability of lass, ( = C) given a luster j, i.e. P( j ). To ompute the lass posterior, first the marginal density of the luster j given lass is onsidered from the joint density of the newly observed data x and the luster as, or, j ) = S x j, x ) dx j ) = j x, ) x ) dx (5) S x t should be noted that j is defined only over the support of newly observed data S x, whih has been expliitly mentioned in the above integrals. n the first term of the integrand of (5), we further make a fair assumption that lustering is onditionally independent of the lass given the data, i.e., j x, ) = j x ). Thus, j ) = S x j x ) x ) dx Sine the image data is disrete, the integral over S x is approximated by the finite sum and the luster onditional is given by, j (6) ) = P( j x ) x ) x X Using (6), the lass posterior an be omputed easily from the Bayes rule, j )P( ) P( j ) = C (7) j )P( ) = Given the lass posterior for a luster, the lass ˆ that maximizes this posterior is hosen as the onsistent lass for that luster. This is equivalent to a MAP seletion of the lass given a luster under the assumption of zero-one loss funtion. This proedure yields the most onsistent lass for eah luster. An impliit assumption has been made that eah luster belongs to one of the lasses in the reognition voabulary. Let K be the set of the luster maps that are onsistent with the lass. n other words, K ontains all those luster maps that yield lass as the MAP estimate of their orresponding lass posterior given by (7). The Constrained Class Density Map is obtained simply by linear weighting of eah pixel density of the Class Density Map by the orresponding pixel probability of the onsistent Cluster Maps, i.e., ons ( xt ) = porig( xt )P( j xt ) j K p (8) where, p orig (..) is the original and p ons (..) is the onstrained lass onditional density. An intuitive explanation of (8) is that the onstrained map is a linearly mixed, weighted density map where weights follow Gaussian distribution in the feature spae beause eah luster map approximately orresponds to a Gaussian distribution. t an be noted, that the final onstrained density map for a given lass tends to enhane those areas in the original density map that are supported by the statistis of the regions in the test image that show strong assoiation with the given lass.. MODEL SELECTON n the proposed generative approah, we need to know the number of the omponents in the Gaussian mixture models in () as well as in (4), whih amounts to the problem of model seletion. The maximum likelihood approah is not appropriate for this task, as it would always prefer more omponents. Several tehniques have been proposed under the topi of model seletion [8]-[]. Full Bayesian model seletion tehniques provide a more prinipled method of model seletion and generally use a parametri or hierarhial form to approximate the prior distribution over the parameters [8]. The BC or MDL approah, as proposed by Rissanen [] an be shown to be asymptotially onsistent version of the full Bayesian model seletion tehniques. n the MDL tehnique, the desription length (DL) is given as, DL = - log X/θ) + (l/2) log n (9) where X is the data, θ is the parameter vetor ontaining all the model parameters, l is the number of parameters and n is the dataset size. The first term in the right hand side of (9) is the negative log likelihood (i.e., ode length of likelihood) and the seond term is ode length of the parameters, whih ats as a penalty term as the number of parameters inreases. However, there are two main limitations while implementing the full Bayesian or MDL riterion. First, they generally need iterative shemes to ompute the model evidene or the likelihood, whih is prone to getting stuk in loal extrema and seond, they are fairly slow and beome almost impratial for the online soft lustering appliations. n the present work, we propose a Kullbak-Leibler Divergene (KLD) based method to estimate the number of omponents in the mixture model based on ertain assumptions. The KL Divergene (KLD) is defined as,

4 94 where p~ ( x ) p~ ( x ) KL ( ~ p p ) = ~ x )log dx () x ) is the true density of data x and x) is the model density. To apply the KLD in the omponent seletion proedure, the model density is given by the mixture of Gaussians, but the true density is not known. We assume the data histogram (empirial distribution) to be the representative of the true density. This assumption is reasonable in the ase of natural sene images, as they are mostly omposed of smooth and homogeneous regions. The true density is further approximated by inrementally fitting Gaussians on the modes of the data histogram. For this purpose, we use seond order normal approximation of the empirial distribution [2]. Parameters of the Gaussian are obtained by mathing the urvature of the Gaussian with that of the empirial density at the mode. This is equivalent to using the inverse of the information matrix at the mode of the empirial density as the ovariane matrix of the Gaussian. The expeted information matrix is given as, J ( x ) = E T log x ) log x ) x x where E is the expetation with respet to x. J(x) is further approximated by its value at the mode xˆ of x), and the stohasti or observed information matrix is given as: ( x ) = log x ) log x ) () x x By using (x), the ovariane matrix C g of the approximated Gaussian is given as, C g = - (x) (2) Beause we are using the disrete empirial distribution as x), omputing (x) in () is equivalent to omputing the negative of the hessian of the best fit urve at the mode of the distribution. The neighborhood of log x) at the mode was assumed to be loally quadrati for omputational purposes. Let Ω( xˆ ) = d nω { x ρ : x ρ R } = ρ be the set of all data points in the neighborhood of the mode xˆ. n the loal neighborhood of xˆ, the log-density an be expressed as: log x ρ T T ) = xρ Hxρ + B xρ + 2 T xˆ (3) where H is the hessian matrix, and B and are other onstants. The hessian an be omputed using Singular Value Deomposition (SVD) on the loal neighborhood of the mode of the histogram. Let us denote the ombined vetor for all the neighbors of xˆ as Y = [log x ),,log xn )], the data matrix as D, eah omponent of whih is a monomial of order two or less, and ϑ as the vetor that ontains all the omponents of H, B, and. The quadrati fitting amounts to sloving the linear set of equations given by Y = Dϑ, whih is solved using SVD as ϑ = (VΣ - U T )Y, where D = UΣV T. n the proposed method, the first estimate of the model density is obtained by fitting the first Gaussian at the mode of the histogram and the KLD between the empirial and the model density is omputed. The integral in () is approximated by the finite sum over the disrete bin data. Next, a new density is omputed by fitting one more Gaussian on the next highest mode of the histogram and mixing the two Gaussians. The mixing parameters have been assumed to be uniform. The modes are seleted to be the maxima in their loal neighborhood of a prespeified size. As per the above formulation, as the number of Gaussians is inreased, KLD dereases and starts inreasing when the number of Gaussians grows beyond that supported by the data distribution. n addition, it should be noted that if the KLD remains stable as the omponents are inreased, it is an indiator that one or more of the previous omponents has already explained the new modes. This tehnique works fairly well in low dimensional, sparse feature spae (as is the ase with the images ontaining natural regions) as finding the modes in the loal neighborhood of a histogram is relatively easy. V. FEATURE EXTRACTON The type of features to be extrated from an image depends on the nature of the sene lassifiation task. n the present work, we deal with the sene images primarily ontaining natural regions. Although not suffiient, low-level features suh as olor and texture ontain good representation power for the region lassifiation of natural senes. A. Color Features Color is an important omponent of the natural sene lasses. However, the olor-based features suffer from the problem of olor onstany. For the natural senes, we argue that given enough variations in the training data set, we an apture the lass olor distribution in varying illumination onditions. n addition, in the outdoor natural senes, the problem of artifiial illuminants is not as restritive as the hanges in brightness. To extrat the olor features, we need to represent the olor in a suitable spae. n the present work, we found that the results with three different spaes i.e., retangular HSV, g- RGB, and Luv were fairly similar. We hose generalized RGB (g-rgb) spae, as it normalizes the effets of brightness variations effetively. The g-rgb spae is given by two R G oordinates, ( r =, g = R + G + B R + G + B ) where (R, G, B) are the oordinates in the RGB spae. t is lear that one degree of freedom is lost in this onversion beause the third Ω T

5 95 oordinate of this spae is simply ( - r - g). This loss of one degree of freedom is true for almost all the olor spaes, whih either normalize or dispense with the luminane information. the seond subsetion ontains the results of the proposed observation-onstrained generative approah applied to real natural senes. B. Texture Features n ontrast to the olor, texture is a not a point property. nstead, it is defined over a spatial neighborhood. Texture an provide good disrimination between natural lasses that resemble in olor domain e.g., water and sky. However, in our ase, texture should be dealt with more are, as any single lass does not have a unique texture. Within a semantially oherent region, there might be areas of high or low textures, with different sales and diretional uniformity. n the lassifiation of suh regions, a very strong texture measure an sometimes undo the good work done by the olor features. Several tehniques have been reported in the literature to ompute the texture in a pixel neighborhood. The famous ones inlude Multiresolution Simultaneous Autoregressive (MSAR) model [3], Gabor Wavelets [4] and the Seond Moment Eigenstruture (SME) [5, 6]. n the present work, we have used a weaker measure of texture yielded by the Seond Moment Eigenstruture (SME), whih an apture the essential neighborhood harateristis of a pixel. The seond moment matrix at eah image pixel (i, j) is given by: M(i, j ) 2 i W W = i j W W i j 2 j (4) A. Model Seletion Results on the Syntheti Data To verify the effetiveness of the proposed KLD based model seletion sheme and ompare the results with the MDL based approah, we applied these tehniques on a simulated dataset ontaining, samples drawn from a mixture of three Gaussians in two-dimensional spae. The mixing parameters used in the mixture model were.7,.2, and.. The KL Divergene based model seletion sheme was applied to the data histogram and the size of neighborhood was hosen to be 3 3. Fig. shows the hange in KLD as the number of omponents is inreased in the Gaussian mixture. The KLD falls sharply when the omponents are inreased from to 3 and then starts inreasing. Thus, this method orretly finds the number of omponents in the mixture data. To ompare the results of KLD with MDL approah, MDL was omputed for the given dataset using (9). The maximum likelihood estimates of the parameters were obtained using EM. Beause EM is sensitive to the initialization, we performed the MDL omputation several times. Fig. 2 and show two typial plots of the hange in desription length as the number of omponents is inreased. n, the MDL metri inorretly favors two omponents, possibly, beause the EM algorithm was stuk in a loal maximum. where k is the gradient of the image in spatial diretion k over the olor hannel for k = i, j and = R, G, B. W is the window over whih these gradients are summed. We use Gaussian weighting in window W around the pixel of interest to give more weight to the pixels near it. The texture obtained from (4) is a olored texture beause the above matrix aptures the texture in olor spae instead of usual intensity spae. The seond moment matrix an be shown to be a modifiation of bilinear, symmetri positive definite metri defined over the 2D image manifold embedded in a 5D spae of (R, G, B, i, j) [6]. The eigenstruture of the seond moment matrix represents the textural properties. Two measures have been defined in [5] using the eigenvalues of the matrix, anisotropy = - λ2/λ, and normalized strength = 2 (λ + λ 2 ), where λ and λ 2 are the two eigenvalues of matrix M(i, j) and λ > λ 2. n the present work, we have used the ombination of anisotropy and the strength alled texture strength (S) as the texture measure. t is given as the produt of anisotropy and normalized strength. V. RESULTS AND DSCUSSON We have divided this setion in two subsetions. The first subsetion disusses the simulation results of the proposed KLD-based model seletion sheme on the syntheti data, and MDL Fig.. KL-Divergene based model seletion on the syntheti data. KLD orretly favors three omponents KLD Number of Components Number of Components MDL Fig. 2. Two typial runs of the MDL based model seletion tehnique on the syntheti data. MDL inorretly favors two omponents. MDL orretly favors three omponents Number of Components

6 96 () (e) Fig. 3. Cluster Maps orresponding to different lusters (j) obtained from the soft lustering of the input image. nput olor image. j = () j = 2 (d) j = 3 (e) j = 4 (f) j = 5 n another typial run when EM yielded the orret parameters aused by onvergene to the global maximum, the MDL metri orretly favors three omponents. Thus, it an be seen that even for a very low dimensional spae, MDL is not robust when EM is not initialized in proximity of the global maximum. n addition, the average time taken by a typial MDL run was about 935 s (in Matlab, 733 MHz) while that by KLD was only.2 s. Thus, we an get substantial savings in terms of speed by using KLD. This is espeially important for the unsupervised lustering in the test image, whih has to be done online unlike the supervised learning. B. Results on Real mages The proposed observation-onstrained generative approah was tested with several images primarily ontaining natural regions. The lass reognition voabulary ontained five lasses: sky, water, skin, sand/soil, and grass/tree. A total of 3 pixelwise labeled images, eah of size pixels, were used as a training set for the supervised learning. An input olor image is shown in Fig. 3. t an be noted that the sene is fairly luttered and ontains regions with varying illumination intensity, e.g., tree regions. Sky pixels in the image show bimodal distribution due to the presene of white textured loud in a blue sky. At first, the soft lustering is performed on the image and the Cluster Maps are generated. For this purpose, the number of lusters in the image is estimated using the KLD based tehnique. Fig. 4 shows the hange in KL Divergene when the number of lusters (K) is inreased. As K is inreased from to 2, sharp (d) (f) derease in KLD an be notied. After K = 5, the KLD almost stabilizes indiating the non-signifiane of further inrements in K. Hene, we have hosen five as the number of lusters in the original image. ntuitively, one an observe five broad ategories in the input image i.e., sky, water, red wall, floor/skin, and tree/grass. One the number of lusters has been determined, unsupervised learning of the mixture model generates the Cluster Maps i.e., P( j x ). These maps for five lusters in the original image are given in Fig. 3 -(f). n the Cluster Maps, a brighter pixel indiates a higher probability of assoiation of that pixel with the given luster. t is lear from Fig. 3 that different lusters have aptured the similarities in the image pixels. nreasing order of j, the maps broadly represent grass, red wall, sky, floor, and water. The semanti lasses, whih represent relatively small regions, have been merged with other lusters, e.g., skin regions were merged with red wall or floor. There are some intuitively obvious inorret assoiations in the luster maps, e.g., for j =, parts of sky, water, and swimming ostume have been lustered along with the grass/tree lass. However, it should be emphasized that at this stage the algorithm has no notion of a semanti lass. The results of probabilisti lustering are purely based on the similarity within the newly observed data. KL Divergene t Number of Components Fig. 4. KLD based omponent seletion for the image from Fig. 3 Fig. 5. For the input image shown in Fig. 3 (a.), Class Density Map orresponding to the lass tree/grass. Corresponding Constrained Class Density Map.

7 97 The main aim of the soft lustering was not to obtain perfet lustering, but to obtain a probabilisti estimate of ohesiveness in various pixels, whih ould later be used for onstraining the results of the lass generative models. These errors in the lustering do not have limiting influene on the final Constrained Class Density Maps. This will be shown in the later part of this setion. As the next part of the proess, generative mixture models were learnt for eah lass in the reognition voabulary. The feature sets orresponding to all the pixels belonging to eah lass (in 3 training images) were used. The number of omponents in eah lass onditional mixture model was learnt using the KLD-based tehnique explained before. n the present results, we fous on the lass grass/tree to explain all the steps involved in the observation-onstrained generative approah. For this lass, the KLD-based model seletion favored five omponents in the mixture model. A Class Density Map displaying x k ) for every pixel in the input Class () Class () () Fig. 6. Posterior distribution P( j ) over lasses () given the different Cluster Maps j from Fig. 3. j =, j = 2, () j = 3, (d) j = 4, (e) j = Class () (e) Class () Class () (d) image (Fig. 3 ) orresponding to lass grass/tree is given in Fig. 5. A brighter pixel indiates a higher density. t an be seen that the generative model has orretly predited high density for the grass/tree regions. However, there are several non-grass areas with signifiant probability of being grass, e.g., parts of water, wet floor et. These areas share the features lose to those of grass/tree in some arbitrary illumination or physial onditions like sale, season et. They have the potential of misleading the Bayesian refinement of lass maps while using improved priors obtained from the sene ontext ues. Fortunately, the statistis of the observed grass/tree pixels in the given test image has enough information to remove these outliers by onstraining the overall generative model. To obtain the Constrained Class Density Maps for lass grass/tree, first those Cluster Maps are found that are onsistent with lass grass/tree. For this purpose, posterior distributions over the lasses given eah Cluster Map are obtained using (7). Fig. 6 displays the posterior distributions for Cluster Maps j = 5 displayed in Fig. 3 (f). Eah plot is a disrete valued graph where horizontal axis displays the five different lasses i.e.,. sky, 2. water, 3. skin, 4. sand/soil, and 5. grass/tree. For eah luster map, the maximally onsistent lass is obtained using the MAP estimate of the posterior. t is lear from the plots that the luster map for j = has muh higher probability of being from the lass grass/tree ( = 5) than the other four lasses. Also, this is the only luster that is onsistent (in the sense of MAP) with the lass grass/tree. This orrespondene between the hosen lass and the Cluster Map is supported intuitively from Fig. 3 and Fig. 5. All other luster maps have natural semantis favoring other lasses in the reognition voabulary. Thus, the set of onsistent luster maps, K for the lass grass/tree has ardinality one. The Cluster Map orresponding to j = (Fig. 3 ) was used to onstrain the original Class Density Map for grass/tree (Fig. 5 ) using (8). The Constrained Class Density Map ontaining pons( xt ) for eah pixel in the input image orresponding to the lass grass/tree is given in Fig. 5. The densities of false grass regions, e.g., parts of wet floor, water et. have been signifiantly redued. n ase, more than one luster would have been onsistent with the lass grass/tree, the onstrained map ould be easily omputed using all the onsistent luster maps in (8). Similar onstrained maps were also obtained for the remaining lasses in the reognition voabulary. To display the results of the probabilisti lassifiation of the image regions in Fig. 3 using the observationonstrained generative approah, we have used the MAP paradigm. Eah image pixel is lassified as belonging to lass i, whih has highest posterior P( x ). Posteriors are omputed using the onstrained lass onditional densities given in (8) and assuming equal priors for all the lasses. The resulting lassifiation is given as binary images in Fig. 7 i t

8 98 (f) where white pixels represent the pixels belonging to the given lass. The overall good diriminative lassifiation results are indiative of good Constrained Class Density Maps. Sky, water, and grass/tree regions have been lassified almost perfetly exept on the edges, beause of the problem of edginess in the texture measure. Use of improved ombinations of the eigenvalues of the seond moment matrix to avoid this problem has been left as a future work. n the training data, the features of the sky and water regions are distributed lose to eah other and lassifiation just based on the training data would be quite misleading. Reasonably good lassifiation results for these regions were obtained using our approah (Fig. 7 and ()), beause the statistis of these two regions in the test image ould reinfore their respetive lass assoiations, generating good Constrained Density Maps. t an be noted that the above MAP based lassifiation sheme assigns eah pixel in the test image to one of the lasses in the reognition voabulary. But some of the image pixels may not belong to any of the lasses in the voabulary, e.g., in the given test image, pixels pertaing to the red wall and assoiated steel gate, hairs et. n suh ases, the MAP sheme assigns those pixels to the best possible lasses in the voabulary. Other minor artifats in the lassified regions are due to the above reason. n future, we plan to expand the reognition voabulary to ontain more lasses. n Fig. 7 (e), the onrete floor has been lassfied as sand beause there is no pereptual differene between the two. This is a good example, whih advoates the use of the sene ontext to disriminate between regions that are almost impossible to disambiguate using purely low level features. Beause there was no exlusive lass for the red wall in the voabulary, it has been lassfied in the semantially nearest lass skin. Similarly, skin and sand regions have been partially mislassified aused by extreme overlap in low-level features. However, it should be noted that MAP-based lassifiation was used purely for evaluation purposes. The main results of the proposed generative approah are the Constrained Class Density Maps for eah lass that are to be further used in the iterative sene ontext generation sheme. The proposed generative method was applied to another image given in Fig. 8. The Class Density Map orresponding to the lass sky has been shown in Fig. 8. High density has been assigned to most of the regions of the sky, although the map shows some regions of water as potential andidates of being sky. The Constrained Class Density Map in Fig. 8 () shows improvement over the original Class Density Map, as the false regions have shrunk signifiantly. However, there are still some non-sky regions that remain andidates for lass sky. A areful investigation of these regions reveals that these are watery regions that are either splashes or refletions in the intermixed region. Based on the olor and texture features, used in the present work, it is almost impossible to differentiate these pixels from those belonging to the bright loudy sky. Thus, if the statistis of the newly observed image also supports the hypothesis obtained from the Class Density Maps, there is little one an do exept either enhaning the feature set, or using some kind of highlevel ontextual ues. V. CONCLUSONS We have proposed and suessfully demonstrated the use of an observation-onstrained generative approah for the probabilisti lassifiation of image regions. A probabilisti approah towards lustering and lassifiation leads to a useful tehnique, whih is apable of refining the lassifiation results obtained using the generative models. The proposed sheme is robust to the errors in the lustering. A KL Divergene-based fast omponent seletion proedure has been proposed for natural sene images. n the future, we intend to work towards evolving effiient shemes to generate distribution over sene hypothesis using the Constrained Class Density Maps. The proposed probabilisti lassifiation forms the first step of a promising feedbak framework to iteratively refine the sene ontext as well as the region hypothesis. ACKNOWLEDGMENT The first author would like to thank Amit Singhal and Zhao Hui Sun from Eastman Kodak Company, Rohester, and Daniel Huber and Goksel Dedeoglu from Carnegie Mellon University, Pittsburgh for their useful suggestions. () (e) Fig. 7. Disriminative lassifiation of image regions. The above maps are binary maps where a white pixel represents the presene of the orresponding lass. nput olor image. -(f) show lass maps for sky, water, skin, sand, and grass/tree respetively (d) (f)

9 99 () Fig. 8. nput olor image. Class Density Map orresponding to lass sky. () Corresponding Constrained Class Density Map. REFERENCES [] A. Torralba and P. Sinha, Statistial Context Priming for Objet Detetion, Pro nternational Conferene on Computer Vision, CCV, 2. [2] A. Vailaya, Semanti Classifiation in mage Databases, PhD thesis, Mihigan State University, 2. [3] T. M. Strat, Natural Objet Reognition, Springer-Verlag, 992. [4] N. W. Campbell, W. P. J. Makeown, B. T. Thomas, and T. Trosianko, The Automati Classifiation of Outdoor mages, nternational Conferene on Engineering Appliations of Neural Networks, pp , June 996. [5] M. Storring, H. Andersen, E. Granum, Skin olour detetion under hanging lighting ondition", H. Araujo and J. Dias (ed.), 7th Symposium on ntelligent Robotis Systems, pp , 999. [6] C. M. Bishop, Neural Networks for Pattern Reognition, Oxford University Press, 995. [7] A. Dempster, N. Laird, and D. Rubin, Maximum Likelihood from nomplete Data Via the EM Algorithm, J. Royal Statistial So., Ser. B, vol. 39, no., pp. 38, 977. [8] D. Hekerman and D. Chikering, A Comparison of Sientifi and Engineering Criteria for Bayesian Model Seletion, Tehnial Report MSR-TR-96-2, Mirosoft Researh, June, 996. [9] M. H. Hansen, and B. Yu, Model Seletion and the Priniple of Minimum Desription Length. JASA, Vol. 96, No. 454, pp , 998. [] C. Farley and A. E. Raftery, How Many Clusters? Whih Clustering Method? Answers Via Model-Based Cluster Analysis, Tehnial Report, No. 329, Department of Statistis, University of Washington, 998. [] J. Rissanen, Modeling by Shortest Data Desription, Automatia, vol. 4, pp , 978. [2] M. A. Tanner, Tools for Statistial nferene, 3 rd Ed., Springer-Verlag n., 996. [3] J. Mao and Jain A. K., Texture Classifiation and Segmentation using Multiresolution Simultaneous Autoregressive Models, Pattern Reognition, vol. 25, no. 2, pp , 992. [4] B. S. Manjunath and W. Y. Ma, Texture Features for Browsing and Retrieval of mage Data, EEE Trans Pattern Analysis Mahine ntelligene, vol. 8, No. 8, pp , 996. [5] C. Carson, S. Belongie, H. Grennspan, and J. Malik, Region-Based mage Querying, CVPR 97, Workshop on Content-Based Aess of mage and Video Libraries, 997. [6] N. Sohen, R. Kimmel and R. Malladi, A General Framework for Low Level Vision, EEE Trans on mage Proessing, Vol. 7, No. 3, pp. 3-38, 998.