PAPER Improved Color Barycenter Model and Its Separation for Road Sign Detection

Transcription

1 IEICE TRANS. INF. & SYST., VOL.E96 D, NO.12 DECEMBER PAPER Improved Color Barycenter Model and Its Separation for Road Sign Detection Qieshi ZHANG a), Student Member and Sei-ichiro KAMATA b), Member SUMMARY This paper proposes an improved color barycenter model (CBM) and its separation for automatic road sign (RS) detection. The previous version of CBM can find out the colors of RS, but the accuracy is not high enough for separating the magenta and blue regions and the influence of number with the same color are not considered. In this paper, the improved CBM expands the barycenter distribution to cylindrical coordinate system (CCS) and takes the number of colors at each position into account for clustering. Under this distribution, the color information can be represented more clearly for analyzing. Then aim to the characteristic of barycenter distribution in CBM (CBM-BD), a constrained clustering method is presented to cluster the CBM-BD in CCS. Although the proposed clustering method looks like conventional K-means in some part, it can solve some limitations of K-means in our research. The experimental results show that the proposed method is able to detect RS with high robustness. key words: road sign (RS) detection, driver assistance system (DAS), color triangle, color barycenter model (CBM), constrained clustering 1. Introduction The research of road sign (RS) detection becomes more and more important for driver navigation. Because it can regulate traffic and indicate the road situation for guidance and warning, which is a crucial part function of driver assistance systems (DAS). For example, if drivers disregard the temporary stop sign or the speed-limit sign, DAS will notice the driver and give an emergency warning to avoid accident happen. However, it is difficult to detect RS from videos directly, due to the unknown changes of driving environment, such as lighting, size, and rotation. The lighting problem is existed in natural environments and cannot be avoided. Usually, it is various dues to the weather changes (rainy, cloudy, smog, haze, sunny, etc.), time of day changes (morning, noon and night etc.) and differences of RS itself (paint color, fade etc.). It affects the purity of color and intensity largely. If the lighting problem cannot be overcome, almost all kinds of methods will be hard to obtain ideal results. The size problem is usually caused by distance. Although the size of RS is standard, it is always changing during driving in visual. For this reason, if only the RS with specific size is detected, it will be useless in practice. The rotation Manuscript received February 19, Manuscript revised June 28, The authors are with the Graduate School of Information, Production and Systems (IPS), Waseda University, Kitakyushu-shi, Japan. a) Q.Zhang@Akane.Waseda.jp b) Kam@Waseda.jp DOI: /transinf.E96.D.2839 problem also happens sometimes. Although the normal position of RS is perpendicular to the trajectory of the vehicle, but as time pass by or the angle of view, the RS may not in the original position or viewpoint. If this problem cannot be solved, some RS will be missed. Therefore, these three kinds of problem should be conquered for RS detecting in natural condition. For the RS can be detected and used in DAS, many researchers have been devoted to solve these problems [1]. In recent years, the color-based approach becomes popular, because it is one major feature and many methods can be desired under different color spaces. In image processing area, RGB color space is widely used, and many color thresholding methods have been used or desired to segment the RS. Based on RGB color space, Estevez and Kehtarnavaz [2] thresholded the redness pixels by the difference of R with the connection of G&B. And de la Escalera et al. [3] calculated the different relations between R, G, and B components to separate RS. Benallal and Meunier [4] let the difference between R&G and R&B channels as two stable features for RS detection. Fang et al. [5] used the neural network (NN) to extract color and shape feature, and Ohara et al. [6] adopted 2 simple 3-layered NN to cluster the color. Bahlmann et al. [7] calculated the color feature by non-linear color transform to seven color representations. After that, our previous work [8] proposed the original color barycenter model (CBM) which called as color barycenter hexagon (CBH) model to analyze the color feature by the RGB color space conversion and improved it in [9], [10]. Tsai et al. [11] projected the RGB color space as the Eigen by principal component analysis (PCA), and used radial basis function NN to train the classifier for detection. Zhang et al. [12] calculated the difference of the R, G, and B vectors to segment the color image into binary for reducing the lighting influence. Greenhalgh and Mirmehdi [13] transformed the original image into a normalized red/blue component image and used support vector machine (SVM) to detect the RS. Besides the RGB color space, the most frequently employed space is the HSI/HSV. Liu et al. [14] presented a pseudo RGB-HSI conversion method without nonlinear transform to extract the color. Paclik et al. [15] presented the approximate formulas for converting the RGB color space into HSI and thresholding it. And de la Escalera et al. presented a strategy to calculate the hue in HSI color space in [16] and also used a non-linear transformation over hue and saturation to enhance the desired color in the image (red and blue) and used the RS model to find the best match Copyright c 2013 The Institute of Electronics, Information and Communication Engineers

2 2840 IEICE TRANS. INF. & SYST., VOL.E96 D, NO.12 DECEMBER 2013 in [17]. Fang et al. [18] proposed a method in HSI color space by using a similarity measure between hue component and previously stored hue values of particular colors in RS. Vitabile et al. [19] presented a sub-space dynamic thresholding technique to find all possible RS candidates according to their special color. And Maldonado-Bascon et al. [20] also analyzed the color feature in HSI color space. Besides RGB and HSI/HSV space, YIQ and YUV also are used for RS detection. However, these kinds of methods are sensitive to the lighting condition. If the lighting is stronger or weaker, the non-adaptive or training based methods will hard to detect the correct color feature. Moreover, the existing color analysis based methods only try to find one component or the relation of two components in corresponding color space, then use the thresholding method to segment it. Thus these methods are hard to detect multi-color, which required, they just detect the color intensity without the purity. Therefore, the weaker color region cannot be detected in complex surroundings sometimes. Although, the previous proposed CBM based segmentation methods [9], [10] can obtain the effective results, they never considered the influence of number with same color and the separation between magenta and blue regions is not accurate in some situations. In addition, sometimes the previous methods cannot obtain the correct result by the limitation of separation method. In this paper, in order to overcome the limitations of the existing color spaces based methods and to detect the color feature more effectively, the CBM is improved. And then, the constrained clustering method is studied to separate the barycenter distribution in CBM (CBM-BD) for obtaining more accurate color separation result. Based on the multi-color separation result, the RS candidates of the original image will be segmented. Then the size, aspect ratio, and color ratio criterions are used to remove the wrong candidates for confirming the RS. After these processes, the RS in different environments can be detected ideally. The flowchart of the improved CBM based RS detection system as shown in Fig. 1. The main contributions of this research are summered as follows: 1. The CBM-BD is covered into cylindrical coordinate system (CCS) as the improvement of CBM for analyzing the color feature more accurate and easier. In our previous research, the CBM-BD is described in polar coordinate system (PCS) [8] and rectangular coordinate system (RCS) [9], [10]. However, although in PCS and RCS with the corresponding segmentation approach can obtain the acceptable result, some shortcomings need be solved. Such as the limitations of color cast and luminance fluctuation etc. in PCS. And the limitations of wrong segmentation of magenta and blue regions, number influence and incorrect segmentation under multi-peak situation etc. in RCS. For solving these limitations, the CBM-BD is improved to CCS, which can describe all color barycenters in a connection and uniformity space. Fig. 1 Flowchart of the proposed method. 2. A constrained clustering method is proposed for separating the barycenter of CBM-BD in CCS more accurate. For data clustering without the training or prior of the sample set, conventional K-means is an ideal approach. However, it has some limitations, such as the initialization problem, shortcoming of non-globular problem, and empty cluster problem etc. If these limitations cannot be solved, the clustering result will meaningless in our research because all problems listed above will be happen in CBM-BD. Different from conventional K-means, the proposed constrained clustering method can overcome the limitations above and obtain the more accurate result with the initialization and two types pairwise constraints. The remaining of this paper is organized as follows: Section 2 introduces the original CBM and its significant. Section 3 introduces the improved CBM in CCS and its separation. In Sect. 4, RS candidates extraction is introduced and some experiments are compared with other state-of-theart methods. Finally, conclude the proposed method and discuss the future works in Sect Original Color Barycenter Model and Its Significant As described in the previous section, for detecting the RS in complex surroundings and separating the color region more accurate, the original CBM is proposed [8]. After that, for improving the separation strategy, the adaptive thresholds selection method is proposed [9], [10]. This section introduces the original CBM briefly, and for more details refer the previous work [10].

3 ZHANG and KAMATA: IMPROVED COLOR BARYCENTER MODEL AND ITS SEPARATION FOR ROAD SIGN DETECTION Color Barycenter Model (CBM) In our research, the three components of RGB color space are transformed into a color triangle in 2D plane. For one point (pixel) in RGB color space, it has a corresponding color triangle, which is created by: 1. Represent red, green, and blue components of pixel s color in original RGB color space with three vectors (R, G, B ). 2. Project vectors R, G, B into 2D plane with 2π/3 angle, respectively. 3. Connect three vector apexes to create the color triangle. After these three steps, the color triangle is created. Then calculate the barycenter of the corresponding color triangle for all colors and the distribution will constitute the CBM. Table 1 Advantage of saturation representation in proposed CBM than HSV. Vertical coordinate is the range of saturation; horizontal coordinate is the corresponding color groups as the color bar shown (16 groups with the 16 colors, respectively). It is easier to conclude that the saturation expression of proposed CBM is more accurate than HSV. 2.2 Significant of CBM Although the RGB color space is widely used in image processing, it cannot show the color information intuitive. For analyzing the color information easily, HSV/HSI color space usually is used instead of RGB. However, in our study the color separation ability of HSV/HSI is not accurate enough. To obtain more accurate color separation result, the proposed CBM is adopted. Normally, one RS is composed with one or two colors of red, blue, and yellow. Therefore, here we only discuss the separation ability of these three kinds of colors. The examples shown in Table 1 indicate the saturation representation ability of CBM is better than HSV, which will helpful for color separation. Generally, the saturation means the purity of color. If the saturation can be reflected by high range from similar colors and accord with visual experience, it will be helpful for color separation. Table 1 uses four examples to show the advantage of proposed CBM. In Table 1 (a), R v and B v are fixed as 0 and 128, G v is changed from 0 to 255 and be divided into 16 groups ordinal as the color bar shown. This example shows that the proposed CBM has an absolute advantage than HSV. Because in HSV if one component equal to 0, no matter other components were changed, the saturation must be 1. Obviously, it is unreasonable. Different from HSV, the proposed CBM can keep the saturation consistent with corresponding color. In Table 1 (b)-(d), three (R v, G v, B v ) combinations have shown the ability of CBM in normal color. From these three examples, we can see that for high purity and brightness color groups, the range of CBM is larger than HSV. Although the range of CBM in the low purity and dark color groups is smaller than HSV, it keeps the visual experience than HSV. By these four examples of Table 1, we can conclude that the proposed CBM can show the color information more accurate and accord with the visual experience than HSV. 2.3 Shortcoming of Previous CBM-BD Approaches Although the original CBM-BD separation in PCS [8] is fast and simple, but the result is not accurate, especially when the color cast happen. For overcoming the shortcoming of PCS, the RCS by adaptive thresholding is proposed [9], [10]. However, it has three shortcomings, in spite of the ideal results obtained in normal condition:

4 2842 IEICE TRANS. INF. & SYST., VOL.E96 D, NO.12 DECEMBER 2013 obtain the color separation result more accurate. 3.1 CBM-BD in CCS Fig. 2 Difference of CBM-BD in different coordinate systems. (a) is artificial image, (b)-(d) is the CBM-BD in different coordinate systems, and the color points in different distribution are the real color of (a). 1. The separation of magenta and blue regions is not accurate sometimes. 2. The influence of the number of barycenter with similar color is ignored. 3. The multi-peak of one color region cannot be separated correctly. For discussing the shortcomings visually, an example is shown in Fig. 2 with the artificial image. In this figure, three circles are used to show above shortcomings which marked as 1, 2, and 3, respectively. Figure 2 (b) is the CBM-BD in PCS, it can reflects the characteristic of color information, but hard to separate it automatic [10]. In (b), for the region 1 and 2 the statistic based separation is correct, but wrong in region 3. Figure 2 (c) is the CBM-BD in PCS, it can obtain more accurate result than PCS, but the shortcomings mentioned above are hard to be solved. For the problem 1, the color of the car should be in one color cluster, however in the RCS it is separated into two regions as shown in region 1 of Fig. 2 (c). For the problem 2, the purity of color in region 2 is high, so it only shows in limited points in RCS. However, the limited points influenced the shape of the boundary curve of CBM-BD in order that the barycenter of color separation result will be wrong. For the problem 3, although the main color of the two signs looks same, its not continuous, so lead to the multi-peak in one color region 3. In this situation, linear analysis based approach hard to obtain correct results. 3. Improved CBM in CCS and Its Separation Although the previous proposed CBM-BD separation approaches in PCS [8] and RCS [9], [10] can obtain acceptable result, they have some shortcomings as shown in above. For overcoming these shortcomings, the CBM-BD is improved to CCS and a constrained clustering method is studied to For calculating the barycenter of CBM and showing in CCS, the first step creates the color triangle of each color and calculate the corresponding barycenter in PCS. After that, convert the barycenter from PCS to RCS, and then convert into CCS based on RCS. Color triangle creation: in order to calculate the barycenter of the color triangle easily, the apexes of color triangle are defined as (R x, R y ), (G x, G y ), and (B x, B y )by Eq. (1): (R x, R y ) = (0, ( R v ) ) (G x, G y ) = 3 2 G v, 1 2 G v ( 3 ), (1) (B x, B y ) = 2 B v, 1 2 B v where the R v, G v, and B v are the original values of pixel color. Barycenter Calculation (in RCS): the color barycenter (C x, C y ) can be calculated by Eq. (2): ( C x = 1 3 (R x + G x + B x = 1 ) G v B v ( ) ( C y = 1 3 Ry + G y + B y = 1 3 Rv 1 2 G v 1 2 B. (2) v) After the calculation of Eq. (2), all possible color barycenters construct a hexagon region. PCS conversion: in order to describe the color barycenter in CBM more easily, it is calculated into RCS by Eq. (3). arctan (C y /C x ), C x > 0, C y 0 π/2, C x = 0, C y > 0 ϕ = arctan (C y /C x ) + π, C x < 0 3π/2, C x = 0, C y < 0, (3) arctan (C y /C x ) + 2π, C x > 0, C y 0 r = C 2 x + Cy 2 where (ϕ, r) is the coordinate of barycenter in PCS, and ϕ (0, 2π], r [0, 85] by the definition of CBM. CCS conversion: for analyzing the color information more accurate and overcoming the shortcomings of RCS, the CBM-BD is converted into CCS. The coordinate is set as (x, y, z), which is calculated by (ϕ, r) ofrcsas: x = sin(ϕ) y = cos(ϕ) z = r. (4) In this case, the CBM-BD is on the surface of the cylinder, although the influence of brightness can be overcome, it cannot reflect the information when the color with larger difference of (R v, G v, B v ) and smaller value of them because of the dimension reduction. Such as (R v, G v, B v ) = (5,10, 60), the intensity of barycenter is stronger, but the color looks like black in the vision. For this kind of colors, it also should

5 ZHANG and KAMATA: IMPROVED COLOR BARYCENTER MODEL AND ITS SEPARATION FOR ROAD SIGN DETECTION 2843 Fig. 3 The CBM-BD in CCS for true color. where k 1, 2,, 6, and ϕ = {π/3, 2π/3,π,4π/3, 5π/3, 2π}. If only consider the color information without the saturation, some barycenter with weak color will be separated into color categories wrongly. Such as, color of (250, 220, 220) is red, but the color information is too weak to reflect the color information by vision. If this kind of color is separated into color categories, it will become to noise not the useful information. For solving this, this kind of barycenter should be clustered in achromatic category, not the color categories. Based on the analysis above, all barycenters will be separated into another one category R 7 independent of the 6 color categories R k above by Eq. (7): x = β sin(ϕ), 0 <ϕ<2π R 7 = y = β cos(ϕ), 0 <ϕ<2π, (7) z = r nor, 0 r nor T be clustered into a achromatic region. By this reason, the brightness information is added to discriminate the color, which has same hue and salutation, but different brightness. The coordinate of CCS (x, y, z) is redefined as: x = β sin(ϕ) y = β cos(ϕ), (5) z = r nor where β is the brightness of the color. It is defined by the minimum value of the three components of RGB space and normalized to [0, 0.5] to fix the diameter as 1. And r nor is the normalized r of RCS to make the CBM-BD in a unit space. After this, all barycenters as shown in the cylinder of Fig. 3. Different with the CBM-BD in PCS [Fig. 2 (b)] and RCS [Fig. 2 (c)], more relevant can be reflected in CCS [Fig. 2 (d)]. In CCS, the adjacent color, which distributed in the near region, will not be separated. And the influence of number and multi-peak problems also can be solved by the clustering method, which studied in the next section. 3.2 CBM-BD Separation in CCS For detecting the RS with color information, a key step is detecting the correct color of RS. Although our CBM-BD in CCS can reflect the characteristic of color clear with color barycenter, it cannot be used for detecting directly. Consider the dominant color of RS is red, blue, and yellow, so if the regions of these kinds of color cannot be separated the color information about RS will be lost. In our research, the color distribution of proposed CBM is fair, the barycenter of all colors is distributed average (it includes 6 regions [10]). If we want to separate the red, blue, and yellow regions, we have to separate green, cyan, and magenta for increasing the accuracy of separation with the same clustering condition. Based on this reason, the CBM-BD should be separated into 6 regions by Eq. (6): R k = x = β sin(ϕ), ϕ k <ϕ<ϕ k+1 y = β cos(ϕ), ϕ k <ϕ<ϕ k+1 z = r nor, 0 r nor 1, (6) where T is the threshold of saturation which used to separate the barycenters of weak color and set T as 10 [10]. Correspondingly, the range of z = r nor in R k (Eq. (6)) is changed into T r nor 1. Based on these definitions, the barycenter of all colors can be separated into 7 categories simply, but not accurate enough. Although the initial separation with the 7 categories can be used to separate the barycenter of color directly, the result is not accurate and the result almost same as PCS. For increasing the separation accuracy and overcoming the influence of color cast, clustering method is adopted to solve this. In our case, the clustering of barycenter is real data based, so the training or prior of sample set based methods cannot be used. For overcoming the number influence and multi-peak problem of previous works, the distribution based method also cannot be used. By analyzing the characteristic of CBM, in this paper we propose a constrained clustering method to solve CBM-BD separation problem in CCS. Consequently, we consider one initialization and two types pairwise constraints: Initialization constraints specify that all cluster centers should be defined. Must-link constraints specify that two instances have to be in the same cluster. Cannot-link constraints specify that two instances must not be in the same cluster. In the following, the processes will be introduced and the constraints will be discussed in details. Firstly, initialize the cluster centers. Separate the whole CBM-BD into 7 non-overlapped categories based on the definition of 7 regions R k, and calculate the centroids of barycenter in the corresponding regions as the initial cluster center c k. If the initial category R k is empty, the cluster center c k will set as the geometrical center of R k. Secondly, define the pairwise constraints. The mustlink constraint defines a relation over the barycenters, if the barycenters in the same region R k as shown in the light red region of Fig. 4 they must be in the same category. The category is defined by Eqs. (8) and (9):

6 2844 IEICE TRANS. INF. & SYST., VOL.E96 D, NO.12 DECEMBER 2013 Fig. 4 Example of pairwise constraints (planform of CCS). Point c 0 is the wait-judgment cluster center, and P A, P B, P C are the points under different conditions. Light red region means the current must-link region of c 0, light gray region means the current cannot-link region of c 0, and light blue regions mean the wait-judgment region of c 0. The color of all points means the real color of corresponding position in CBM. R k = x = β sin(ϕ), ϕ k δ ϕ <ϕ<ϕ k+1 + δ ϕ y = β cos(ϕ), ϕ k δ ϕ <ϕ<ϕ k+1 + δ ϕ, (8) z = r nor, 0 r nor T + δ t where k = {1, 2,, 6}, δ ϕ = π/12, δ t = 5 [10] and when k = 7theR 7 is defined as: R 7 = x = β sin(ϕ), 0 <ϕ<2π y = β cos(ϕ), 0 <ϕ<2π. (9) z = r nor, 0 r nor T + δ t The cannot-link constraint defines a relation over the barycenters, if the barycenters not in the same region R + k as shown in the light gray region of Fig. 4 they must not be in the same category. It is defined by Eqs. (10) and (11): R + k = x = β sin(ϕ), ϕ k + δ ϕ <ϕ<ϕ k δ ϕ + 2π y = β cos(ϕ), ϕ k + δ ϕ <ϕ<ϕ k δ ϕ + 2π,(10) z = r nor, 0 r nor T + δ t where k = {1, 2,, 6}, and when k = 7theR + 7 is defined as: R + 7 = x = β sin(ϕ), 0 <ϕ<2π y = β cos(ϕ), 0 <ϕ<2π z = r nor, T + δ t r nor 1. (11) After these two constraints R k and R+ k of K categories, only the region R + k R k needs to be judged for separation. For describing this clear, an example is shown in Fig. 4. In this example, point c 0 is the current cluster center and another three points show three types of barycenter under different conditions. Although point P B is the nearest point of c 0, by the cannot-link constraint the two points must not be in the same cluster. Instead, P A is the fastest point of c 0,by the must-link constraint the two points must be in the same cluster. Only point P C, which in the overlapped region needs be calculated for clustering. Lastly, for each barycenter in CBM-BD calculate the minimum distance from it to each cluster with the pairwise constraints. Only if the barycenters do not satisfy the constraints needs be judged. In this case, if the barycenter is closest to its own cluster, leave it where it is. And if the barycenter is not closest to its own cluster, move it into the closest cluster. Repeat above steps until a complete pass through all barycenters results in no barycenter moving from one cluster to another. In our CBM-BD, we define the cluster center c k cannot move out the corresponding region R k. When the 7 overlapped regions are separated into 7 nonoverlapped categories based on the above steps, the final clustering result can be obtained. The pseudo code of this produce is given in Algorithm 1. Although the main structure looks like conventional K-means, the key parts are different and shown in the gray background parts of the Algorithm 1. Besides, the proposed constrained clustering method can solve some limitations of conventional K-means in our research as follows: 1. Fixed number of clusters can make it difficult to predict what K should be. We set K = 7 for fixing the number

7 ZHANG and KAMATA: IMPROVED COLOR BARYCENTER MODEL AND ITS SEPARATION FOR ROAD SIGN DETECTION 2845 of categories. 2. Different initial centers can result in different clusters. We use centroids of barycenter to fix the cluster center. 3. Does not work well with non-globular clusters. We use the pairwise constraints to overcome this. 4. Wrong with empty clusters. The geometrical center is used to instead of the cluster center and allows it empty. 5. Time consuming. In our experiment, conventional K-means needs about 2.5s for one image. Different with it, the proposed constrained clustering method only needs about 70ms. Based on the advantage above, the proposed constrained clustering method can obtain more accurate result than con- Table 2 Robustness of color separation. ventional K-means. 3.3 Advantage of CBM-BD in CCS than PCS and RCS In this section the advantages of proposed CBM-BD in CCS than previous works [8] [10] are as follows: 1. High robustness of color separation: Table 2 shows the robustness of CBM-BD in CCS than PCS and RCS. The example of robustness has been given in Fig. 2, the improved CCS not only conquer the shortcoming of PCS, but also keep the robustness of PCS. 2. High accuracy of segmentation: Fig. 5 gives an example to show the segmentation accuracy in PCS, RCS, and CCS visually. Obviously, with the different barycenter distribution and separation approach, the CCS and its separation approach can maintain the dominant hue of RS with less noise than PCS and RCS. 4. Experimental Results and Discussion To evaluate our method, experiments were performed on the real data collected from different conditions. All experiments are tested using Matlab on Mac PC with 2.4GHz CPU and 4Gb memory. The image size is , and takes about 70ms. To certify the effectiveness of our methods, other five methods are compared. And for comparing the segmentation and detection result fair, only the segmentation approach of all methods is used. The candidate extraction method is same, which introduced in Sect RS Candidates Extraction Based on the above strategy of color region segmentation, the RS candidates can be detected. After that, some simple geometric property based filtering is used to extract the real RS. As the RS can be circular, rectangular or triangular, the following three criteria are adopted: Size criterion: it requires the area of RS candidates in the range as Eq. (12) A min w RS h RS < Area < A max w RS h RS, (12) where Area is the area of candidate, w RS and h RS are the width and height of RS, respectively. By analyzing several different sizes of RS in test video, we found that if the size of candidate smaller than 144 pixels, it is difficult to Table 3 Percentage of color in different types of RS. Fig. 5 Accuracy of segmentation. Input is the original color image. CBM-BD is the corresponding barycenter distribution in PCS, RCS, and CCS. Separation is the result of clustered CBM-BD with the corresponding approach. Segmentation is the result of selected RS candidates with the separated CBM-BD.

8 2846 IEICE TRANS. INF. & SYST., VOL.E96 D, NO.12 DECEMBER 2013 Fig. 6 Example of RS detection in night condition ( , 84 frames). Fig. 7 Example of RS detection in sunny condition ( , 18 frames). Fig. 8 Example of RS detection in rainy condition ( , 37 frames). Fig. 9 Example 1 of RS detection in normal condition( , 55 frames). Fig. 10 Example 2 of RS detection in normal condition ( , 60 frames). be observed. And no matter how long the distance is, the size of RS should not be larger than 2500 pixels. Based on this prior, the minimum size ratio coefficient A min is set as and maximum size ratio coefficient A max is set as Through this filtering, the small regions, such as the signal lamp, color mark of some objects, etc., and the large region, such as the colored building, symbol, etc. will be removed.

9 ZHANG and KAMATA: IMPROVED COLOR BARYCENTER MODEL AND ITS SEPARATION FOR ROAD SIGN DETECTION 2847 Table 4 Detection accuracy under different conditions. No. of RSs Proposed Ref. [3] Ref. [9] Ref. [10] Ref. [11] Ref. [20] Ref. [21] Night Condition % 63.7% 71.1% 80.7% 67.4% 76.3% 57.0% Sunny Condition % 73.0% 77.7% 87.2% 79.7% 83.1% 64.9% Rainy Condition % 73.5% 79.7% 91.2% 81.4% 82.3% 63.7% Normal Condition % 78.8% 83.8% 93.8% 82.0% 84.2% 69.8% Table 5 Detection inaccuracy under different conditions. No. of RSs Proposed Ref. [3] Ref. [9] Ref. [10] Ref. [11] Ref. [20] Ref. [21] Night Condition % 14.8% 8.2% 6.7% 9.6% 11.9% 12.6% Sunny Condition % 18.9% 10.1% 8.1% 13.5% 12.2% 15.5% Rainy Condition % 17.3% 11.9% 7.9% 15.5% 11.5% 17.7% Normal Condition % 12.3% 8.1% 6.2% 11.8% 11.9% 13.4% Fig. 11 Some false examples. Aspect ratio criterion: it requires the RS candidates to satisfy the aspect ratio as: ( hrs min, w ) RS > AR, (13) w RS h RS where AR is the aspect ratio, and for circular, rectangular, and triangular, the ideal ratio is AR = 1, but in fact it always has some shift. For deciding AR threshold, we select 50 real detected RS randomly and get the statistics to set AR as Based on AR, more noise and wrong candidates can be eliminated. However, some regions, which have the same size and shape of real RS, are selected by mistake. Therefore, the judgment of color ratio is used to solve this problem. Color ratio criterion: it calculates the color percentage in the standard RS block. Table 3 shows some of them as an example. Based on the color ratio listed in this table, we consider the influence of colorcast and fade. Here let the offset error with 15% of standard RS be accepted. The 15% is calculated by the detection precision. And if one of the criteria is satisfied, it will be accepted by the following color percentage: Red [31.6%, 65.1%], Blue [20.9%, 68.5%], and Yellow [23.4%, 45.8%], where the range is calculated by the number of different colors with an offseterrorofall types of standard RS. After these criteria judgment, the real RS candidates will be extracted. 4.2 Experimental Results Comparison and Discussion In this section, several experiments are tested to verify the efficiency and robustness of the proposed method. In the experiment, the comparison in different situations, such as night condition (Fig. 6), sunny condition (Fig. 7), rainy condition (Fig. 8), and normal condition (Figs. 9 and 10) is analyzed. Results show that although the color information about RS is not strong sometimes, the RS can be detected correctly based on the color feature detection with CBM in blurred and weak light conditions. In these figures, some reference methods are used for comparison and the results are shown in Tables 4 and 5. Table 4 shows the accuracy of detection results, which means the existed RS is detected correctly. Table 5 shows the incorrect results, which means the RS is not existed but detected by mistake. By these examples, the results show that the proposed method is robust in lighting, size, and rotation. For the lighting problem, the strong point is the robust segmentation of goal region because the proposed method analyzed and distilled the color relation to instead of the light intensity. For the size problem, due to the CBM can keep more pixels of the RS region with less noise, so the small region also can be detected. And for the rotation problem, the proposed method also can be used to detect the RS correctly with the color relation and aspect ratio. By these comparisons, we can conclude that the proposed CBM based method can detect the RS more accurate than other reference methods. Like all methods have defects, our method also has some limits. In some situations, the proposed method cannot obtain the correct results due to the color information and the completeness of shape is not strong enough. Here we summarize the problems as four points and shown as the red circle in Fig. 11. First one is due to the reflection

10 2848 IEICE TRANS. INF. & SYST., VOL.E96 D, NO.12 DECEMBER 2013 of light will let the RS incomplete or color cast. Such as the example shown in Fig. 11 (a), the color of yellow turn to green. Second one is the pure of color too thin, it includes too light and dark. Such as the example shown in Fig. 11 (b)(c), it hard to be detected due to the RS have limited color information (too bright in (b) and too dark in (c)), although it can be detected by eyes easier. Third one is incomplete. By the adopted candidates extraction methods, if the RS is incomplete it will be removed. Figure 11 (d) shows another missed example, which the center part is covered. Sometimes, the RS similar object will be detected wrongly as shown in Fig. 11 (e). By analyzing the success and false examples, if the color and color region can be detected completely, the RS will be correctly detected. 5. Conclusion and Future Works In this paper, our novel and original CBM is improved to CCS for RS detection. The color barycenter distribution in CCS can keep the neighbor relation for analyzing the characteristic easier. And a constrained clustering method is studied to separate the color distribution more accurate for RS detection. Experimental results show that the proposed method can obtain excellent performance under different environments, such as the lighting changes, complex backgrounds, color, etc. However, if the color of RS is too thin or the region of RS is not complete, the RS may be missed. For overcoming this problem, some training and prior knowledge based estimation strategy will be studied. References [1] V. Kastinaki, M. Zervakis, and K. Kalaitozakis, A survey of video processing techniques for traffic applications, Image Vis. Comput., vol.21, no.4, pp , April [2] L. Estevez and N. Kehtarnavaz, A real-time histographic approach to road sign recognition, Proc. IEEE Southwest Symp. on Image Analysis and Interpretation, pp , April [3] A. de la Escalera, L.E. Moreno, M.A. Salichs, and J.M. Armingol, Road traffic sign detection and classification, IEEE Trans. Ind. Electron., vol.44, no.6, pp , Dec [4] M. Benallal and J. Meunier, Real-time color segmentation of road signs, Proc. IEEE Canadian Conf. Electrical and Computer Engineering (CCECE 03), vol.3, pp , May [5] C.Y. Fang, S.W. Chen, and C.S. Fuh, Road-sign detection and tracking, IEEE Trans. Veh. Technol., vol.52, no.5, pp , Sept [6] H. Ohara, I. Nishikawa, S. Miki, and N. Yabuki, Detection and recognition of road signs using simple layered neural networks, Proc. Int l Conf. on Neural Information Processing (ICONIP 03), vol.2, pp , June [7] C. Bahlmann, Y. Zhu, R. Visvanathan, M. Pellkofer, and T. Koehler, A system for traffic sign detection, tracking, and recognition using color, shape, and motion information, Proc. IEEE Intelligent Vehicles Symp. (IV 05), pp , June [8] Q. Zhang, J. Zhang, and S. Kamata, Color barycenter hexagon model based road sign detection, Proc. Int l Multi-Conf. of Engineers and Computer Scientists (IMECS 08), vol.1, pp , March [9] Q. Zhang and S. Kamata, Automatic road sign detection method based on color barycenters hexagon model, Proc. Int l Conf. on Pattern Recognition (ICPR 08), pp.1 4, [10] Q. Zhang and S. Kamata, A novel color descriptor for road-sign detection, IEICE Trans. Fundamentals, vol.e96-a, no.5, pp , May [11] L.W. Tsai, J.W. Hsieh, C.H. Chuang, Y.J. Tseng, K.C. Fan, and C.C. Lee, Road sign detection using eigen colour, IET Computer Vision, vol.2, no.3, pp , Sept [12] K. Zhang, Y. Sheng, and J. Li, Automatic detection of road traffic signs from natural scene images based on pixel vector and central projected shape feature, IET Intelligent Transport Systems, vol.6, no.3, pp , [13] J. Greenhalgh and M. Mirmehdi, Real-time detection and recognition of road traffic signs, IEEE Trans. Intelligent Transportation Systems, vol.13, no.4, pp , [14] W. Liu, X. Chen, B. Duan, H. Dong, P. Fu, H. Tuan, and H. Zhao, A system for road sign detection, recognition and tracking based on multi-cues hybrid, Proc. IEEE Intelligent Vehicles Symp. (IV 09), pp , June [15] P. Paclk, J. Novovicova, P. Pudil, and P. Somol, Road sign classification using laplace kernel classifier, Pattern Recognit. Lett., vol.21, no.13-14, pp , [16] A. de la Escalera, J.M.A. Armingol, and M. Mata, Traffic sign recognition and analysis for intelligent vehicles, Image Vis. Comput., vol.21, pp , [17] A. de la Escalera, J.M. Armingol, J.M. Pastor, and F.J. Rodrguez, Visual sign information extraction and identification by deformable models for intelligent vehicles, IEEE Trans. Intelligent Transportation Systems, vol.15, no.2, pp.57 68, June [18] C.Y. Fang, C.S. Fuh, P.S. Yen, S. Cherng, and S.W. Chen, An automatic road sign recognition system based on a computational model of human recognition processing, Comput. Vis. Image Understand., vol.96, no.2, pp , [19] S. Vitabile, G. Pollaccia, G. Pilato, and E. Sorbello, Road signs recognition using a dynamic pixel aggregation technique in the hsv color space, Proc. IEEE Int. Conf. Image Analysis and Processing (ICIAP 01), pp , Sept [20] S. Maldonado-Bascon, S. Lafuente-Arroyo, P. Gil-Jimenez, H. Gomez-Moreno, and F. Lopez-Ferreras, Road-sign detection and recognition based on support vector machines, IEEE Trans. Intelligent Transportation Systems, vol.8, no.2, pp , June [21] N. Kehtarnavaz and A. Ahmad, Traffic sign recognition in noisy outdoor scenes, Proc. IEEE Intelligent Vehicles Symp., pp , Sept Qieshi Zhang is a Ph.D. student of the Graduate School of Information, Production and Systems, Waseda University, where he received the M.S. degree in He received the B.E. degree major in Automation and minor in Computer and Application from the Faculty of Automation and Information Engineering, Xi an University of Technology, China in From 2004 to 2006, he was an assistant engineer of the Department of Mechanical Electronically Technology in Xi an Siyuan University, China. From 2010 to 2012, he is Research Fellow of the Japan Society for the Promotion of Science (JSPS). His current research interests are in image processing, image compression, image detection, image enhancement and computer vision. Mr. Zhang is a student member of the IEEE and IEICE.

11 ZHANG and KAMATA: IMPROVED COLOR BARYCENTER MODEL AND ITS SEPARATION FOR ROAD SIGN DETECTION 2849 Sei-ichiro Kamata received his M.S. degree in computer science from Kyushu University, Japan, in 1985, and his doctor of engineering degree from the department of Computer Science, Kyushu Institute of Technology, Japan, in From 1985 to 1988, he was with NEC Ltd., Kawasaki, Japan. In 1988, he joined the faculty at Kyushu Institute of Technology. From 1996 to 2001, he was an associate professor in the Department of Intelligent Systems, Graduate School of Information Science and Electrical Engineering, Kyushu University. Since 2003, he has been a professor at Graduate School of Information, Production and Systems, Waseda University. In 1990 and 1994, he was a visiting researcher at the University of Maine, Orono. His research interests are image processing, pattern recognition, image compression, remotely sensed image analysis, space-filling curves and fractals. Prof. Kamata is a member of the IEEE, and the ITE in Japan.