Traffic Sign Segmentation and Recognition in Scene Images

Traffic Segmentation and Recognition in Scene Images Fei Qin 1, Bin Fang 1, Hengjun Zhao 1 1. Department of Computer Science, Chongqing University, Chongqing 400030, China E-mail: jiemy@cqu.edu.cn, fb@cqu.edu.cn, zhhj@cqu.edu.cn Abstract: Traffic s provide drivers with very valuable information about the road, in order to make driving safer and easier. They are designed to be easily recognized by human drivers mainly because their color and shape are very different from natural environments. Automatic traffic sign detection and recognition is important in the development of unmanned vehicles, and is expected to provide information on road signs and guide vehicles during driving. This paper deals with traffic sign detection and recognition from image sequences. In order to reduce the computational complexity in the scene image processing, an effective method for traffic sign segmentation based on color distance is proposed in this work. The scene image is mapped to a matrix by computing the color distance. Through the selection of appropriate distance threshold on the basis of large number of scene image samples, it quickly obtains the binary image. To obtain better classification performance, we use linear support vector machine with the Distance to Border features of the segmented blobs to get the shape information, and then realize the rough classification based on Color-Geometric Model. Traffic sign classification is implemented using RBF kernel based support vector machine with edge related pixels of interest as the feature. The experimental result shows that our method can work well and achieve the traffic sign segmentation and recognition with sufficiently high processing speed and satisfactory accuracy. Key Words: Traffic ; Image Segmentation; Color Distance; Pattern Recognition; Distance to Border; SVM Algorithm 1. INTRODUCTION Computer vision plays an important role nowadays along with advancement in science and technology. Automatic Traffic Recognition (TSR), as an important subtask of Intelligent Transportation System (ITS), has been of great interest for many years. How to realize traffic sign recognition under complex conditions in real-time has been drawing technical attention but remaining to be a challenge [1]. Generally speaking, a pattern recognition system consists of two major components, detection and classification. Traffic sign detection is the premise of traffic sign recognition, and how to locate fast and detect accurately the traffic sign is very crucial in real-time system. How to identify the traffic sign rapidly and precisely is equally fundamental. Like many other countries, in China, traffic signs are designed to be easily detected and recognized by human beings according to the colors and shapes. Color is considered as one of the most distinct and useful features in traffic sign segmentation as well. However, it is not easy to identify whether a pixel is red or blue or not, as pictures captured by cameras are affected by various factors, such as weather, illumination, and so on. Selecting a proper color space plays a crucial role in color segmentation. Usually a color image is converted from the RGB (Red, Green and Blue) color space to HSV (Hue, Saturation and Value) color space because the hue value is invariant to the illumination as well [2]. Yet the present transformation algorithm spends much time on floating-point operations which results in low efficiency. Hence in our work, we focus our attention on color segmentation based on the RGB color space. A wide variety of different color segmentation schemes for traffic sign images have been proposed in recent years, and some good results have been achieved. Aryuanto Soetedjo et al. [3] started with selecting an appropriate threshold based on a great of experiments. A pixel is red if R/(R+G+B)>TR, where R, G, B, TR represent respectively the value of R, G, B of the pixel and the selected threshold. Another color segmentation strategy proposed by Mudan Hu, et al. [4] is based on Chromatic Aberration (CA) in RGB color space. They selected λr-g-b as the characteristic operator for the red prohibition sign and λb-r-g for the blue directional sign. Taking the red sign as an example, CA is 0 if λr-g-b<0; CA is 255 if λr-g-b>255; CA equals to λr-g-b otherwise. CA represents the gray value of chromatic aberration. The Otsu s thresholding algorithm is subsequently induced and applied to output segmented image. Several research works about traffic sign classification have been done and certain results have been achieved. Merve Can Kuş et al. [5] developed a colored traffic sign recognition system using Scale Invariant Feature Transform (SIFT). Min Shi et al. [6] discussed two kinds of features for representing each sign and analyzed four different kernels and two SVM types. King Hann Lim et al. [7] proposed a hybrid traffic sign recognition scheme combining of knowledge-based analysis and radial basis function neural classifier. In this work, we present a general framework for detection and recognition of traffic signs from image sequences. This paper is organized as follows: Section 2 details our proposed method for traffic sign segmentation. Section 3 focuses on the technique for traffic sign recognition. Section 4 shows the experimental results, followed with the conclusion and discussion in section 5. 978-1-4244-7210-9/10/$26.00 2010 IEEE

2. TRAFFIC SIGN SEGMENTATION Traffic s are designed to be easily detected and recognized by human beings according to the distinct colors and shapes. Similar to many other countries, in China, traffic signs of road safeties are mainly classified into three categories, the prohibition signs, directional signs and warning signs. Red is the basic color for prohibition signs, with background mainly in white. In the directional signs, blue is the background color, mainly with white-core designs. As for warning signs, yellow is chosen as the fundamental color, with black border, and the core-based designs are also mainly in black. Taking the traffic signs in our country as the example, as shown later, we will details out method in this section, covering three steps: the color distance, the selection of threshold and the thresholding segmentation. 2.1 Color Distance As we know, Euclidean distance is defined as the straight line distance between two points. In N dimensions, the Euclidean distance between two points is: d( p, q) = ( p q ) where p i (or q i ) is the coordinate of p (or q) in dimension i. Euclidean distance is one of the most common uses of distances in Bayes Decision Theory and Statistical Pattern Recognition. Assume that the feature space is isotropic and the problem of feature scaling does not exist; it can be used to measure the similarity between two vectors. Logically, the RGB values of each pixel can be considered as a point in the RGB color space. Therefore, the Color Distance (CD) [8], which denotes the difference between two colors, is defined as: N i= 1 i i 2 (1) Similarly, for the blue directional signs, a pixel is identified as blue if CD blue <TH blue and the yellow warning signs, a pixel is yellow if CD yellow <TH yellow, where TH blue and TH yellow indicate the blue threshold and the yellow threshold respectively. Unfortunately, we have to pay attention to the selection of Standard Color (SC). It is assumed that the SC of traffic sign is close to the ideal one. So far, we can do the segmentation as long as the right threshold is selected. How to choose an appropriate threshold remains to be a challenge. We take the red prohibition sign as an example in the subsequent subsections. 2.2 Threshold Selection The selection of threshold is crucial. Inappropriate value means introducing excessive amount of noise or taking away the target object, and possibly resulting in failing to extract the target image. Our method of threshold selection is mostly similar to the idea proposed by Ling Luo and Xiying Li [9]. The threshold is not an experience threshold but a statistic threshold. The method to search for color segmentation threshold uses consecutive images which includes traffic signs. Using experience threshold roughly segments these consecutive images. After that, positive and negative samples are classified manually. The next task is to do a statistical work both in positive-sample-set and negative-sample-set. Obviously, our threshold parameter is color distance. We prepare 240 positive samples and 240 negative samples for the statistical work, and the results are shown in Fig. 1. The histogram s abscissa represents value of Color Distance, whose range is from 0 to 255 3. The ordinate represents the number of pixels. CD( i, j) = ( R R ) + ( G G ) + ( B B ) 2 2 2 i j i j i j (2) where R i, G i, B i and R j, G j, B j, respectively, indicate the RGB values of pixel i and j in RGB color space. Obviously, CD(i,j), the distance between color i and j, can be used to measure the similarity between the two colors. According to the color distance model given above, the color distance between a pixel and the standard red (255,0,0) can be written as: CD = ( R 255) + ( G 0) + ( B 0) red 2 2 2 (3) where R, G, B are the values of red, green, and blue of the pixel. Here, we mention the property that CD red represents the prospect of a pixel to be taken as red (as well as non-red) pixel. The lower the value, the higher (lower for non-red ones) is the possibility. Thus, the red prohibition signs, a pixel in traffic sign image is classified into red if CD red <TH red, where TH red represents a red threshold which will be discussed in the next subsection. Fig. 1: Positive-sample-set and negative sample-set s color distance histogram statistics We are focused on the two black broken lines. The positive samples are mostly on the left side of the second broken line, while the negative samples are mostly on the right side of the second broken line. This means that the positive samples and negative samples are mostly able to separate from each other though the color distance. Meanwhile, we also mention that there are hardly any

negative samples on the left side of the first broken line. Between the two broken lines, the closer to the first broken line, the less noise is introduced and the more target is missed; Otherwise, the opposite. Hence, an appropriate threshold can be selected naturally. However, the brightness varies a lot between the images. It is noticed that the average pixel value is less than 100 in too dark images and greater than 180 in too light images [10]. According to our statistical work, the TH value may be selected as 184, 158, 150 for too dark images, normal brightness images, and too light images, respectively. 2.3 Traffic Segmentation Traffic sign segmentation is very vital in the area of traffic sign detection, by which the noise may be attenuated further in order to reduce the computational complexity, and thus improve the detection efficiency and accuracy. Now that the TH value is selected, a binary image is quickly obtained with the following formula: 255, CD < TH Bi = (4) 0, otherwise where Bi represents the value of a pixel in the binary (gray) image. According to the TH values selected for different cases and the rules as stated above, we achieve the binarization under different brightness as shown in Fig. 2. Respectively, Fig. 2(d) shows the binary image of Fig. 2(a), Fig. 2(e) is the binarization result of Fig. 2(b), and Fig. 2(f) comes up with the result of binarization of Fig. 2(c). of each pixel in the image, as it is known that the function f(x)= is monotonous on [0, ). (a) (c) Fig. 3: Warning and directional sign segmentation results (a), (c) Original scene images (b), (d) Segmentation results of our method The next stage is to locate traffic sign candidates. Detected objects can be screened based on some common features, such as area, height to width ratio, position, etc. Traffic sign segmentation becomes very easy based on the binary image. Meanwhile, the shape and color can be utilized to assist with traffic sign recognition. 3. TRAFFIC SIGN RECOGNITION (b) (d) 3.1 Color-Geometric Model (CGM) (a) (b) (c) According to analysis of the color and shape attribute of 116 Chinese traffic signs, it is noted that there is a unique and fixed relation between the geometric shape and the color of traffic signs, which is called Color-Geometric Model (CGM) [11]. Fig.4 shows the unique and fixed relation. Red Blue Yellow (d) (e) (f) Fig. 2: Segmentation results under different brightness (a) Too dark image (b) Normal brightness (c) Too light image (d) Result of (a) (e) Result of (b) (f) Result of (c) In order to further verify the feasibility and applicability of our method, we also select a yellow warning sign image and a blue directional sign image for experimenting purposes. Fig. 3 presents the segmentation effect. We have to point out that, in order to avoid floating-point operations so as to improve the efficiency, it is not necessary to do the square root operation while computing the CD value Prohibition Directional Fig. 4: Framework of CGM Warning We can directly divide the 116 Chinese traffic signs of three major classes into several subclasses by color and shape. It establishes a good foundation for the further recognition.

3.2 Distance to Border (DtB) In order to recognize the shape of traffic sign, many methods have been proposed for obtaining the feature vectors. In this work, we use Distance to Border (DtB) [12] vector as the feature vector for training SVM. DtB is the distance from external edge of the blob to its bounding box. Thus for a segmented blob we have four DtB vectors for left, right, top and bottom. The main advantage of this method is its robustness to translation, rotation and scale. The algorithm is invariant to translation because it does not depend on the position of appearance of the blob in the scene. It is invariant to rotation because all blobs have been previously orientated in a reference position using the DtB vectors. In order to make the DtBs invariant to changes of scale, the blobs are zoomed to 36 36. Therefore, there are four DtB vectors of 36 components for a single instance. Fig. 5 shows the four DtB vectors for a segmented circular blob. (a) (b) (c) Fig. 5: (a) Segmented circular blob (b) Binary image of (a) (c) DtB for circular blob 3.3 Support Vector Machine (SVM) SVM is a set of related supervised learning methods used for classification and regression. It is based on mathematical foundations of statistical learning theory, which was proposed first by Vapnik in 1992 [13]. The main idea is to construct a hyper-plane as the decision surface in such a way that the margin of separation between positive and negative samples is maximized. SVM algorithm is described as follows: Suppose the set of training samples T=(x 1, y 1 ),,(x l, y l ), where x i R n, i=1,,l, y i {1,,k} is the class of x i and k is the number of classes. The two-class classification problem is the optimum equation as follow: 1 ij T ij ij ij T min ( w ) w + C ( ) ij ij ij ξt w (5) w, b, ξ 2 t where w is a normal vector, which is perpendicular to the hyper-plane. The constant C is only as an additional constraint on the Lagrange multipliers. ξ i is a non-zero penalizing function. We get the decision function, that is, the classifier that can decide the class of x. 1, gx ( ) > 0 f( x) = 1, others (6) l g( x) = yak( x, x ) + b i= 1 where x is the input vector to be classified, x i (i=1,,l) is a support vector, and K(x, x i ) is known as kernel function. A kernel constructs an implicit mapping from the input space into a feature space, and then a linear machine is trained in the feature space to classify input vectors. Four different kernels are commonly used: linear kernel: K(x i, x j )=x i T x j polynomial: K(x i, x j )=(γx i T x j +r) d, γ>0 radial basis function (RBF): K(x i, x j )=exp(-γ x i -x j 2 ), γ>0 sigmoid: K(x i, x j )=tanh(γx i T x j +r) where γ, r, and d are kernel parameters. The traditional SVM is a binary classifier. Two common methods to build the multiclass SVM are where each classifier distinguishes (i) between one of the labels to the rest (one-versus-all) or (ii) between every pair of classes (one-against-one). Classification of new instances for one-versus-all case is done by a winner-takes-all strategy, in which the classifier with the highest output function assigns the class (it is important that the output functions be calibrated to produce comparable scores). For the one-against-one approach, classification is done by a max-wins voting strategy, in which every classifier assigns the instance to one of the two classes, then the vote for the assigned class is increased by one vote, and finally the class with most votes determines the instance classification. 3.4 Traffic Classification i i i (7) In the shape classification stage, we need to build a multiclass SVM to classify 5 types of shapes including circle, rectangle, positive triangle, inverse triangle, and octagon. One-against-one classifier is applied here. k(k-1)/2 SVM classifiers perform k-category classification. In case of tie, positives outputs of SVMs are computed to decide the shape and if the total number of votes is lower than 2, the object is discarded like noise. In fact, there are only one octagonal sign and only one inverse triangular sign in China. Once the shape information is obtained, we can realize rough classification based on CGM. Moreover, it also can be utilized to extract Pixels of Interest (POI) by using a mask image corresponding to the shape of the blob. The shape classified blob is first normalized to a size of 80 80. The masking operation is applied, with only those pixels that are crucial part of the sign reserved. Then the edge of the masked POI is extracted and used as the feature vector for traffic sign classification. The detailed process can be found in [14][15]. Different one-versus-all SVMs classifiers with a RBF kernel are employed. The parameters of cost (C) and gamma (g) are also important for the training and testing, we can get the optimal parameters by grid search and cross-validation. We have to point out that both training and testing are done based on the CGM so as to reduce the complexity.

To build the multiclass SVMs, we use a package called LIBSVM that developed by Lin Chih-Jen [16]. 4. EXPERIMENTAL RESULTS In our experiment, train and test sequences have been recorded with a video camera (JAI BB-141 GE) fixed onto the top of a vehicle while driving at usual speed. The video sequences were saved in JPG file format with a resolution of 800 600. We use a dataset of 1000 images which contains 2059 signs in all for the simulation experiment. Table 1 summarizes the experimental results of traffic sign recognition. True Positive (TP) denotes the number of traffic signs correctly classified. False Positive (FP) indicates the number of signs wrongly classified. False Negative (FN) is the number of noise objects wrongly classified as traffic signs. Table 1: Traffic sign recognition results Traffic s No. TP FP FN Red Circle 938 794 97 56 Red Inv-Triangle 43 39 0 0 Red Octagon 32 27 1 4 Blue Circle 475 389 64 41 Blue Rectangle 249 196 23 47 Yellow Pos-Triangle 332 263 39 24 5. CONCLUSION AND DISCUSSION In this work, we present a system for segmentation and recognition of traffic signs in scene images. A color based segmentation method using color distance is introduced. In recognition stage, we develop a linear SVM for shape classification and another SVM with RBF kernel for recognition of POI, where CGM is applied as prior knowledge. Experiment results show that our algorithm is succinct and conducive to real-time processing. However, the selected TH values in this paper depend on the large number of samples captured in special environment. How to dynamically select the right TH value requires further study, and will be explored in the next research phase. ACKNOWLEDGMENTS This work is supported by Natural Science Foundation of China (90820306). REFERENCES [1] D.M. Gavrila. U. Franke, C. Wohler, and S. Gorzig. Real-time vision for intelligent vehicles, IEEE Instrumentation & Measurement Magazme, 4(2), pp. 22-27, 2001. [2] C.Y. Fang, C.S. Fuh, and P.S. Yen. An automatic road sign recognition system based on a computational model of human recognition processing. Computer Vision and Image Understanding, vol. 63, no. 3, pp. 237-268, 2004. [3] Aryuanto Soetedjo and Koichi Yamada. An Efficient Algorithm for Traffic Detection. Journal of Advanced Computational Intelligence and Intelligent Informatics, vol. 10, no. 3, pp. 409-417, 2006. [4] Mudan Hu, Shuangdong Zhu, and Ken Chen. An Effective Method for Traffic Segmentation, International Conference on Intelligent Human-Machine Systems and Cybernetics, pp. 180-184, 2009. [5] Merve Can Kuş, Muhittin Gökmen, and Şima Etaner-Uyar. Traffic Recognition using Scale Invariant Feature Transform and Color Classification. International Symposium on Computer and Information Sciences, pp. 1-6, 2008. [6] Min Shi, Haifeng Wu, and Hasan Fleyeh. Support Vector Machine for Traffic s Recognition, IEEE International Joint Conference on Neural Networks, pp. 3820-3827, 2008. [7] King Hann Lim, Li-Minn Ang and Kah Phooi Seng. New Hybrid Technique for Traffic Recognition. IEEE International Symposium on Intelligent al Processing and Communication Systems, pp. 1-4, 2009. [8] M. Abdullah-Al-Wadud and Oksam Chae. Region-of-Interest Selection for Skin Dectection Based Applications. Proc. the IEEE International Conference on Convergence Information Technology, pp. 1999-2004, 2007. [9] Ling Luo and Xiying Li. A method to search for color segmentation threshold in traffic sign detection. International Conference on Image and Graphics, pp. 774-777, 2009. [10] J.Torresen, J.W.Armingol, and M.A.Salchs, et al. Road Traffic Detection and Classification. IEEE Conference on Intelligent Transportation Systems, pp. 652-656, 2004. [11] Zhu Shuangdong and Liu Lanlan. Color-Geometric Model for Traffic Recognition. International Workshop on Intelligent Systems and Intelligent Computing, pp. 2028-2032, 2006 [12] S. Lafuente Arroyo, P. Gil Jimenez et al. Traffic Shape Classification Evaluation I: SVM using Distance to Borders. Proceedings of IEEE Intelligent Vehicles Symposium, pp. 557-562, 2005. [13] V. Vapnik, Statistical Learning Theory, New York: John Wiley & Sons, 1998. [14] S. Maldonado Bascon, S. Lafuente Arroyo et al. Road- Dectection and Recognition Based on Support Vector Machines. IEEE Transactions on Intelligent Transportation Systems, vol. 8, no. 2, June 2007. [15] Kiran C.G., Lekhesh V.Prabhu, Abdu Rahiman V. and Rajeev K. Traffic Detection and Pattern Recognition using Support Vector Machine. International Conference on Advances in Pattern Recognition, pp. 87-90, 2009. [16] Chih-Chung Chang and Chih-Jen Lin, LIBSVM: A Library for Support Vector Machines, 2001. [Online]. Available: http://www.csie.ntu.edu.tw/~cjlin/libsvm.