Components 0.5. Hue. c c c c R G B R

Transcription

1 ROBUST COLOUR IMAGE SEGMENTATION Philippe Pujas Marie-Jose Aldon Laboratoire d'informatique, de Robotique et de Microelectronique de Montpellier LIRMM - UMR Universite Montpellier II/CNRS 161 rue ADA, Montpellier Cedex 5, FRANCE phone: ; fax: pujasp@lirmm.fr, aldon@lirmm.fr, URL: Abstract In this paper, we present a new method for segmenting a colour image into homogeneous regions. This method uses the discriminant properties of the three perceptual colour attributes : the hue H, the saturation S and the value V. In a preliminary step, each pixel is classied according to a chromaticity degree which characterises its belonging to a colour space zone whose chromatic components are meaningful. In a second step, we apply a segmentation algorithm based on a recursive directed region grower, and which provides a hierarchical scene representation. Two examples are given where an indoor scene and a subsea image are processed to yield uniform regions. We compare the results obtained by using the dierent chromatic/achromatic classication methods described in this paper. Key Words : image processing, colour, segmentation, neural network. 1 Introduction One of the main goal of image analysis is to isolate regions that are likely to come from a single object. By comparison to grey level images, colour images contain three times as much data. Thereby, their use allows to get a much more robust segmentation toward lighting conditions and a better accuracy with regard to the extracted regions [12]. Let us emphasise the fact that colorimetric data make easy intra-scene matching in case of partially occluded objects [2, 11], and inter-scene matching [9] for analysing an image sequence or a pair of stereoscopic images. We have focused our work on the extraction of connex and uniformly coloured regions. The main applications are objects detection and localisation with or without an a priori knowledge on colour, shape extraction for pattern recognition, inter-scene matching for mobile robot vision. The new segmentation approach we present in this paper is based on the idea that the hue is the most discriminant attribute for most of regions, while the intensity and the purity of the colour can be aected by the illumination conditions and by the surface reectance. In the rst part of the paper, we describe the properties of the perceptual (H; S; V ) space in order to justify the choice of this colour representation. We show that this model allows to separate the intensity and chromatic components. Then, we introduce the concept of chromaticity degree and we propose dierent methods to estimate this degree. The last part of the paper is dedicated to the recursive guided region growing algorithm that we have developed to achieve the segmentation process. We present results obtained with inside and outside real scenes. Their comparison with results provided by more classical segmentation approaches illustrates the eciency of the proposed method. 2 Choice of a perceptual colour space 2.1 Introduction A lot of psychophysiologic studies have shown that colour information is generally tri-variant. So, three variables are necessary to describe the whole colour sensation. One can say that colour information is a vector eld of R 3 [4].

2 The Commission Internationale de l'eclairage (CIE) has standardised the frames that allow to split up one colour. It has suggested a physico-psychophysiologic and natural frame : the (R; G; B) frame. In this frame, each colour component corresponds to the intensity of the incident ray in a given band. Unfortunately the (R; G; B) frame does not match the feeling perceived by the human brain. From a perceptual point of view, colour can be described by three attributes : The intensity V represents the amount of light received by the sensor. It depends on the lighting conditions and on the light source emissivity. The hue H is a value which represents the main colour of the pixel in the (R; G; B) triplet provided by the camera. It can be described by an arbitrary value associated to the name of the colour (table 1). Hue : H Colour Red 1 Yellow 2 Green 3 Cyan 4 Blue 5 Magenta Table 1: Some correspondences between the name of the colour and the value of hue in the (H; S; V ) frame The saturation S describes the pureness of the colour : high, pale,... So, the saturation marks the dierence between the red and that we call pink. For instance, high red corresponds to a high value of the saturation while pale pink corresponds to a small one. 2.2 The (H; S; V ) frame All the perceptual frames are dened by a nonlinear dieomorphism. The (H; S; V ) representation is obtained according to the following equations : V = sup (R; G; B) S = 256 H = 8 >< >: V? inf (R; G; B) V G? B V? inf (R; G; B) 2 + B? R V? inf (R; G; B) 4 + R? G V? inf (R; G; B) if V = R if V = G if V = B In these equations we suppose the dynamic of the image signal to be 256. (1) This coordinates transform requires 2N multiplications and 2N divisions that can be tabulated. It is simplest than most of the other perceptual representations. For instance the (Lu? v? ) transform [8] requires 2N divisions, 19 N multiplications, 2 N root extractions and N arc tangent computations : L = 116:4 3p :299R + :587G + :114B? 16 u? = 13L 2:428R + :696G + :84B 5:92R + :117G +? :21 5:262G v? = 13L 2:691R + 5:283G + 1:26B 5:92R + :117G +? :461 5:262G (2) H = arctan v? u? S = p u?2 + v?2 2

3 We can also compare (H; S; V ) to the linear transforms that require 9 N multiplications in all. Table 2 presents the computation times, on a Sun Sparc 2, for applying these transforms to a 256x256 colour image. Space Computation time (ms) Linear 15 HSV 33 Lu? v? 165 Table 2: Computation time for colour space transform for a 256x256 RGB image. (Computed on a Sun Sparc 2) 2.3 Hue periodicity Let consider the feeling of colour produced by a combination of two among the three vectors of the (R; G; B)frame and let c R ; c G and c B be the three normalised components. Their values can be considered as the ratio of red, green and blue in the colour. Let assign the following constraints : c R + c G + c B = 1 (3) c R c G c B = We have represented on gure 1 the value of the three ratios, and the hue resulting of an arrangement of two components. Components 1.5 red yellow green cyan blue magenta red c c c c R G B R Y Hue Figure 1: Colour obtained by combining two vectors of the (R; G; B) frame. In that way a ratio of 5% of red and green ( point Y ) feels like yellow. So we obtain all the perceptible colours whose hue varies in the interval [ ; 6[. We can see that to mix blue and red we must duplicate red after blue. So, the hue corresponding to lim H!6 H feels like red; its value is zero. By another way, equation 1 shows that hue is a periodic variable. Indeed, when V = R, hue belongs to the interval [?1 ; +1], when V = B it belongs to [3 ; 5]. So, with this coordinate transform, hue belongs to the interval [?1 ; 5]. In order to obtain a value belonging to [ ; 6[ we must consider H modulo 6. To avoid a discontinuity in its representation, hue is generally modelled by an angle. So the polar representation is the more adapted, because hue varies continuously from blue to red and green (gure 2). However, the periodicity of hue arises new problems when we have to compute a hues dierence or a hues weighted average. i) Hues dierence Let consider that the hues H 1 and H 2 of two points in a (H; S; V ) colour space satisfy the condition: 3

4 2 green 1 yellow H cyan 3 red blue 4 5 magenta Figure 2: Hue represented with polar coordinates H 2 2 δ 2 H 1 H 1 δ 4 4 H 2 Case 1 Case 2 Figure 3: Dierence of hues (H 1 ; H 2 ) 2 [ ; 6[ 2 (4) The colour dierence of these points is : = (H 1 ; H 2 ) = H 2? H 1 if jh 2? H 1 j 3 = H 2? H 1? 6 sgn(h 2? H 1 ) if jh 2? H 1 j > 3 (5) These two cases are respectively illustrated by the two examples of gure 3. ii) Hues weighted average Let G be the weighted average of two hues H 1 and H 2 respectively weighted by n and 1. Under condition 4, and if n >, we can dene G by : B(H 1 ; H 2 ) = nh 1 + H 2 n + 1 = nh 1 + H 2? 6 n + 1 if jh 2? H 1 j 3 if jh 2? H 1 j > 3 These two cases are respectively illustrated by the two examples of gure 4 (6) 4

5 2 H(1) 2 G H(n) 1 2 H(n) 1 G 4 4 H(1) 2 Case 1 Case 2 Figure 4: Weighted average of two hues 2.4 Low saturation eect Now, we consider the real case where colour is obtained by combining three vectors of the (R; G; B) frame. We call c 1 ; c 2 ; c 3 the three colour components satisfying c 1 + c 2 + c 3 = 1 and such as c 1 c 2 c 3. If c 1 corresponds to the red component, the colours feels like red, and the dierences c 1? c 2 and c 1? c 3 express the deviation from the red. So, if c 2 correspond to the green component, hue will feel like yellow, except if c 2 = c 3. In this case the colour feels like red, but like a less pure red, that we call pink. We can represent the hue H by the polar angle of point E on a chromatic circle, E being the weighted average of the (R; G; B) components, weighted respectively with c R ; c G and c B (gure 5). The distance from E to the circle center increases with the colour saturation. G.2 E H R.7.1 B Figure 5: H; S representation for the point (:7; :2; :1) of the (R; G; B) space Look at the limit case where E is the circle center( c 1 = c 2 = c 3 ). The saturation is equal to zero and the hue, which corresponds to the polar angle of E, is undened. Such a conguration of the colour characterises an achromatic point of the image. We can establish the existence of achromatic and chromatic areas by considering equation 1: S = () sup (R; G; B) = inf (R; G; B) () R = G = B () H = so, H is undened 5

6 So, there exists an achromatic area in the (R; G; B) colour space, where hue is ill-dened. This area corresponds to a low level of saturation. It is a fundamental problem to nd colour components which are suited for the segmentation process. One way to get such information is to identify a chromaticity degree for each pixel. This estimation allows to dene chromatic and achromatic areas in the image. 3 Chromatic and achromatic areas Previous research works [1, 15] conrm that hue is generally the most discriminant attribute in the (H; S; V ) colour space. So, hue can be used for an ecient segmentation of colour images which does not include achromatic areas. However, when the scene is viewed under bad lighting conditions, achromatic regions appear in the image. For instance : under a weak illumination, objects look like black; with a strong lighting, sensors are saturated and the colour distorsion induces a decrease of the saturation value. In such conditions, the hue attribute H is meaningless, and the value V becomes the discriminant attribute for segmenting the achromatic region. In order to optimise the segmentation algorithm and to improve its eciency, it is necessary to dene a way for classifying pixels into a chromatic or an achromatic zone. Considering equations 1, when V = R, we obtain : We can write: H = G? B V? inf (R; G; B) = (G? B) V S H H (G? B) jg? Bj + V V + S S Equation 7 shows that when S is low, a small inaccuracy S involves a large uncertainty H H problem happens with the value V, but in this case S is low (equation 1). (7). The same We must notice that the pixel classication depends only on S and V. The uncertainty H H on H (equation 7). does not depend In a previous paper [1] we have proposed a classication method based on a binary thresholding of the S and V attributes. We present here a more general approach based on the concept of chromaticity degree c(p ) of a pixel P. We can write c(p ) as a function of S and V with c(p ) 1. P is surely chromatic () c(p ) = 1 P is surely achromatic () c(p ) = P is rather chromatic () c(p ) > :5 P is rather achromatic () c(p ) < :5 The identication of c(p ) is done by manual classication of a representative set of pixels. We present two identication methods : 1. Estimation of a piecewise constant splitting line s(p ) and of a constant width transition zone (gure 6). 6

7 s(p ) = if v = 2 and (s > 175) (2 < v < 8) and (s = 175) v = 8 and (17 < s < 175) (8 < v < 17) and (s = 17) v = 17 and (55 < s < 17) (17 < v < 19) and (s = 55) v = 19 and (4 < s < 55) (19 < v) and (s = 4) = +1 if P belongs to the half-space that contains the origin =?1 if P belongs to the half-space that does not contain the origin c(p ) = 1 + s(p ) tanh ( d ) 2 (8) where d is the distance to the splitting line and is the half-width of the transition zone. Chromaticity degree Saturation Value 255 Figure 6: Piecewise constant splitting line. 2. Neural network learning (gure 8). The network we use is described on gure 7. All the neurons are identical. Let i 1 ; i 2 ; : : : ; i n be the inputs weighted by w 1 ; w 2 ; : : : ; w n. The network output is : o = The learning process leads to the following network : nx k=1 w k i k. c(p ) = tanh ( :65796 tanh ( 1:71148 S + 1: V +?1:8858 ) + : tanh ( :79647 S + : V +?:64687 ) + 2:67968 tanh ( 5: S +?: V +?1:22951 ) + 2: tanh ( 2:38767 S + 5: V +?3: ) +?1:844935) (9) 7

8 S V Bias c(p) Figure 7: Neural network architecture. Chromaticity degree Saturation Value Image segmentation Figure 8: Neural network chromaticity degree. 4.1 Overview of the classical segmentation methods Among earlier works related to image segmentation problem, we may distinguish three kinds of approaches which are based on : edge detection [1, 5, 6], region growing [13, 1, 3] and histogram analysis [7, 14, 1]. Edge detection methods are generally based on a local analysis of the image signal variations in order to extract the lines which represent the boundaries of the regions. The major inconvenient of these methods is that they require a preliminary model of the world to allow the grouping of the extracted edges into coherent objects. Region growing algorithms use a global methodology which consists in selecting an image pixel around which the region will expand. The pixel choice must be done according to an optimal criterion, and sometimes this choice may require a preliminary edge extraction. This problem of pixel selection is avoided with split and merge algorithms. Here the original image is divided into basic regions. Adjacent patches are merged into one region if they satisfy a similarity criterion. Results are strongly dependent on the initial splitting and on the merging criterion. Histogramming technics have been applied to the segmentation of colour images. They are based on the assumption that the histogram of the features values (R; G; B) or (H; S; V ) includes well-separated and 8

9 strong peaks corresponding to the image homogeneous regions. The image is segmented by successive thresholding using upper and lower bounds of selected peaks. Then, connected regions must be extracted in the thresholded binary images. We use a region growing algorithm to segment the chromatic (or achromatic) parts of the image into a set of uniform regions. In order to avoid some drawbacks of the classical region growing methods, we have developed a recursive directed algorithm. 4.2 Recursive directed growing method Our algorithm uses a hierarchical data representation. The original data structure is a nondirected strong and weighted graph in which : each vertex represents a pixel and is labelled by the associated (H; S; V ) triplet, a weighted link joins two connex pixels ; the weight associated to each link represents the inter-pixel likeness. The segmentation process adds an upper level to the original graph. This level denes the identied regions. Arcs are created to link each pixel to one region. The region growing algorithm include three steps : During the pre-processing, the graph is built and the inter-pixel likeness is computed for each link. During the second step seeds are selected and regions grow around. The last step consists in rejecting or merging unsignicant regions Graph computation For each pair of connex pixels (P 1 ; P 2 ) we compute the inter-pixel likeness l ip (P 1 ; P 2 ). We have shown that a colour image provides two kinds of information. The chromatic information is obtained by projecting the colour on the Hue axis. The achromatic information is the projection of the colour on the V alue axis. The inter-pixel likeness is a normalised \distance" which depends on a hue distance (equation 11) and on a value distance (equation 12), according to the chromaticity or the achromaticity of the pixels (equation 14). First, we compute the inter-pixel chromaticity degree ip and the inter-pixel achromaticity degree ip. ip and ip belong to the interval [ 1], so they can be considered as fuzzy values. ip (respectively ip ) indicates how much the pair of pixels (P 1 ; P 2 ) is achromatic (respectively chromatic). We have chosen the following denitions : ip = min(c(p 1 ); C(P 2 )) (1) ip = min((1? C(P 1 )); (1? C(P 2 ))) Then we have to compute the chromatic and achromatic likeness : l c ; l a. The value of these functions is near of zero (or one) when the likeness is small (or big). We have chosen : Where k = arccosh(2) l c = cosh?1 (k (H P 1 ; H P2 ) ) P H (11) l a = cosh?1 (k jv P 1? V P2 j ) P V (12) P H = is the maximal hue distance for similar chromatic pixels P V = is the maximal value distance for similar achromatic pixels 9

10 In order to get a more perceptual segmentation one can supersede l a by the following expression : l a = cosh?1 (k jlog(v P 1 )? log(v P2 )j ) (13) P V because the human eye has a logarithmic response. Now we compute the inter-pixel likeness : Growing process l ip = ip l a + ip l c (14) The region growing algorithm is based on the computation of a membership degree. Each region is characterised by the means of hue, saturation and value. The membership degree d b depends on an inter-pixel likeness l ip and on a region-pixel likeness l rp. The region-pixel likeness is similar to the inter-pixel likeness. It uses a region-pixel chromaticity and a region-pixel achromaticity dened in the same way : and rp = min(c(r); C(P )) (15) rp = min((1? C(R)); (1? C(P ))) So l c = cosh?1 (k (H R; H P ) ) R H (16) l a = cosh?1 (k jv R? V P j ) R V (17) Where l rp = rp l a + rp l c (18) k has the previous value R H = is the maximal dispersion of H in the region R V = is the maximal dispersion of V in the region The choice of the membership degree depends on the segmentation objectives : d b = l rp provides a maximal colour deviation. d b = l ip provides a maximal colour gradation. d b = M in(l ip ; l rp ) provides a maximal colour deviation and gradation. The pseudo code for our recursive directed algorithm is : 1

11 While unclassied pixels exists : Choose a Seed S ; randomly for instance R := + S ;initiate the region model with S C := S ; the Current pixel is set to S recursive-directed-function(c; R) End while Function recursive-directed-function(pixel C, region R) For each unclassified pixel in the neighbourhood of C N := the pixel for which l ip (N; C) is the biggest d b := min(l rp (R; P ); l ip (N; C)) If d b > :5 R := R + N C := N recursive-directed-function(c; R) End if End for each End Function Over-merging step Once the region growing algorithm is achieved, an over-merging process can be used to improve the segmentation result. Due to bad choice of some seeds or to noise in the original image small regions appear. The over-merging process merges these regions into a bigger one that satises the connectivity constraint. The process looks around the small regions and classies the neighbour regions according to an inter-region likeness l ir computed in the same way that l ip (equation 14). For all region to over-merge O B := l ir := For all P; P 2 O For all N, in the neighbourhood of P If N 2 T et T 6= O If l ir (O; T ) > l ir B := T l ir := l ir (O; T ) End if End if End for all End for all For all P; P 2 O B := B + N Suppress O End for all End for all 4.3 Experimental results The algorithms described above are illustrated by the segmentation results presented on gures 9 to 12. We have chosen two kinds of real scenes which represent respectively a structured indoor environment (gure 9-a) with a lot of polyhedral and planar objects and a natural outdoor scene where objects cannot be described by simple geometric shapes (gure 9-b). In the segmented images, we have attributed the same grey level to pixels belonging to the same homogeneous region. The rst image (gure 9-a) shows a corridor whose ground coating provides light reections on the ground and on the walls. Some coloured objects have been placed along the walls. 11

12 Computation time (s) Classical Binary Piecewise constant Neural net Graph Computation Growing Merging Total Table 3: Computation time for the recursive directed region growing algorithm (corridor scene). Tests on a Sun Sparc 2 Computation time (s) Classical Neural net Graph Computation Growing Merging Total Table 4: Computation time for the recursive directed region growing algorithm (red shes scene). Tests on a Sun Sparc 2 Figure 1-a presents the results obtained with a classical segmentation method (using an Euclidean distance criterion in the (H; S; V ) space). Here, chromatic and achromatic areas are not split and the same decision criterion is applied to each image pixel during the segmentation process. Figure 1-b displays the regions obtained by classifying pixels according to a binary chromaticity degree. It takes the value 1 (or ) when the pixel belongs to a chromatic area (or not). In the segmented image we can see false regions (reects on ground,... ) which are identied when pixels are near the chromatic/achromatic boundary. These regions disappear when using a chromaticity degree c(p ) such that : c(p ) 1. The results presented on gure 11-a and 11-b correspond respectively to the estimation of c(p ) by a piecewise constant splitting line and by a neural network learning technic. Similar segmentation results are provided by these two solutions. Moreover the neural network method takes advantage of the suppleness of the learning process. The segmentation results for the natural scene presented on gure 9-b, have been obtained with a classical segmentation method (gure 12-a) and with the neural network classication method (gure 12-b). They show that the algorithm can be applied successfully to a large variety of colour images token with dierent lighting conditions. 5 Conclusion In this paper we have proposed a new approach to perform colour image segmentation. The basic idea is a preliminary pixel classication which takes into account its chromaticity degree. This classication allows to select the discriminant attributes which will be used during the segmentation process. In order to group pixels belonging to homogeneous regions, we have implemented a recursive directed region growing method which takes advantage of a hierarchical data representation. The experimental results presented in this paper show the interest of the pixel classication. They illustrate the performances of the dierent algorithms we have implemented to estimate the pixel chromaticity degree. It is obvious that such a classication may be applied to improve other classical segmentation methods like, for instance, histogram thresholding technics [1] References [1] M. Chapron. A new chromatic edge detector used for color image segmentation. In 11 th IAPR International Conference on Pattern Recognition, The Hague, volume 3, pages 311{314, [2] J. Crisman and C. Thorpe. Color vision for road following. SPIE Mobile Robots III, 17:175{184, [3] J. Liu and Y.H. Yang. Multiresolution color image segmentation. IEEE Trans. on PAMI, 16:689{7, [4] R. Machuca and K. Philips. Application of vector elds to image processing. IEEE Trans. on PAMI, 5(3):316{329,

13 (a) (b) Figure 9: Black and white view of the corridor scene (a) and of the red shes scene (b). (a) (b) Figure 1: Regions obtained on the corridor scene with a classical colour segmentation process (a) and with a binary chromaticity degree (b). [5] R. Nevatia. A color edge detector. In 3rd International Joint Conference on Pattern Recognition, pages 829{832, [6] R. Nevatia. A color edge detector and its use in scene segmentation. IEEE Trans. on Systems, Man, and Cybernetics, 7:82{826, [7] R. Ohlander, K. Price, and D. Reddy. Picture segmentation using a recursive region splitting method. Computer Graphics and Image Processing, 8:313{333, [8] Y. Ohta, T. Kanade, and T. Sakai. Color information for region segmentation. Computer Graphics And Image Processing, 13:222{241, 198. [9] M. Okutomi, O. Yoshizaki, and G. Tomita. Color stereo matching and its application to 3d measurement of optic nerve head. In 11 th IAPR International Conference on Pattern Recognition, The Hague, volume 3, pages 59{513, [1] P. Pujas and M.J. Aldon. Segmentation des images couleur : une methode perceptuelle. In 3rd International Conference on Interface to Real and Virtual Worlds, pages 285{294, Informatique'94 Montpellier. [11] F. Sandt and D. Aubert. Comparison of color image segmentations for lane following. In SPIE Mobile Robot VII, Boston, pages 1{12, [12] Y. Shirai. 3d computer vision and applications. In 11 th IAPR International Conference on Pattern Recognition, The Hague, volume 3, pages 236{245,

14 (a) (b) Figure 11: Regions obtained on the corridor scene with c(p ) 1 (a) and with the neural network (b). (a) (b) Figure 12: Regions obtained on the red shes scene with a classical criterion (a) and with the neural network (b). [13] R. Taylor and P. Lewis. Color image segmentation using boundary relaxation. In 11 th IAPR International Conference on Pattern Recognition, The Hague, volume 3, pages 721{724, [14] S. Tominaga. Color image segmentation using three perceptual attributes. In CVPR, pages 628{63, [15] D. Tseng and C. Chang. Color segmentation using perceptual attributes. In 11 th IAPR International Conference on Pattern Recognition, The Hague, volume 3, pages 228{231,