Developing an intelligent sign inventory using image processing

icccbe 2010 Nottingham University Press Proceedings of the International Conference on Computing in Civil and Building Engineering W Tizani (Editor) Developing an intelligent sign inventory using image processing Yichang (James) Tsai Georgia Institute of Technology, USA Abstract Roadway signs are assets important for roadway safety and traffic regulation. However, sign inventory data collection is time-consuming, costly, and sometimes dangerous. This paper presents the research results from our research project, sponsored by the National Academy of Science (NAS) NCHRP Innovation Deserving Exploratory Analysis (IDEA) program, to develop a sign detection algorithm using image processing. The sign detection algorithm is crucial for developing an intelligent sign inventory system. It is technically challenging to develop a generalized algorithm for detecting more than 670 types of signs specified by Manual on Uniform Traffic Control Devices (MUTCD). A generalized sign detection algorithm is developed based on color, shape, texture, and sign location Probability Distribution Function (PDF). The developed sign detection algorithm can eliminate the data collection efforts of reviewing the images containing no signs. This paper briefly presents the developed sign detection algorithm. The experimental test uses 1,105 actual video log images provided by the City of Nashville to validate the developed algorithm. Results show that the developed algorithm is very promising in reducing the manual review effort which can save time and money for roadway infrastructure data collection. Keywords: sign inventory, image processing 1 Introduction Collecting roadway infrastructure data, including traffic signs, such as stop signs and speed limit signs, is essential for state and local transportation agencies to plan, design, construct, operate, and manage transportation assets. Traffic signs are vital for roadway safety, and inventorying them is necessary for compliance with the Manual on Uniform Traffic Control Devices (MUTCD). However, the data collection process is time-consuming and costly. Current software reviews one image at a time, so extracting sign information from the millions of images is still time-consuming and hinders effective data collection. To remedy the problem of reviewing images frame by frame, there is a need to develop algorithms that can batch-process video log images and support an intelligent sign inventory and management system. Some algorithms are designed to handle traffic signs with specific shapes, such as rectangles and triangles (Ballerini et al., 2005; Zhu et al., 2006). Algorithms have been developed to detect and recognize traffic signs under unfavorable conditions (de la Escalera et al., 2003; Yang et al., 2003). Other algorithms have been developed to detect and recognize specific sign types, such as stop and speed limit signs (Tsai and Wu, 2002; Wu and Tsai, 2005; Wu and Tsai, 2006). Although these algorithms have been developed for automatically detecting and recognizing

some particular signs (e.g. stop signs and speed limit signs), they are not suitable for a comprehensive sign inventory because the algorithms are not generalized, and they are unable to recognize the more than 670 types of traffic signs on U.S roadways, a technically challenging job. Figure 1 shows an example in which a speed limit sign (25 mph) in a video log image was detected and recognized. Figure 1, Traffic sign data inventory using image processing algorithms This paper presents the research results obtained from our research project, sponsored by National Academy of Science (NAS) NCHRP Innovation Deserving Exploratory Analysis (IDEA) program, on developing a generalized sign detection algorithm using image processing. In this research project, two innovative, modularized algorithms, sign detection and sign recognition, are developed. They form a solid foundation for developing an intelligent sign inventory and management system. A twostep sign inventory data collection process is proposed to seamlessly incorporate these two algorithms for batch processing millions of video log images, which can save great amounts of time and significant costs. The generalized sign detection algorithm, the first step in the intelligent sign inventory and management system, is developed to reduce the efforts of reviewing the images containing no signs. The generalized sign recognition algorithm, the second step in the intelligent sign inventory and management system, is developed to extract sign attributes, such as sign type, MUTCD codes, etc. This paper focuses on sign detection. The paper is organized as follows. First, the developed sign detection algorithm is briefly described. The experimental test, using the actual video log images provided by City of Nashville, is then conducted and results are presented. Finally, conclusions and recommendations are made. 2 A generalized sign detection algorithm This section briefly presents the developed sign detection algorithm with a special focus on presenting the features/models that are selective to support sign detection. To detect the more than 670 types of signs, the proposed sign detection algorithm uses several features, including sign color, shape, location PDF, and other sign features, such as width-to-height ratio. Feature extraction is important for sign detection. Traffic signs have dominant color, shape, texture, and other attributes, which makes them different from the background. According to the MUTCD standard, traffic signs can have the following ten MUTCD colors: black, blue, brown, green, orange, red, white, yellow, fluorescent yellow-green (FYG) and fluorescent pink (FP). Also, traffic signs are shaped as triangles, rectangles, pentagons, octagons, circles, and crosses. Video log images, which were collected by state Departments of Transportation (DOTs) by using a survey vehicle, show that the traffic sign has the 2

non-uniform, dominant location distribution in an image plane. For example, a traffic sign doesn t appear on the left bottom and right bottom parts in the image. Also, other attributes, such as size, width-to-length ratio, etc. can be used. The following briefly presents the features used for sign detection and the details for sign color feature extraction and shape feature extraction as presented in Tsai et al., 2009. 2.1 Color feature extraction Color is a very important feature for a traffic sign, since sign colors usually receive more attention from the drivers. However, actual sign color may vary because of different lighting, the camera used, and other imaging conditions. For example, the red colors for the same stop sign will have different Red, Blue and Green (RGB) values under different lighting conditions. As a result, sign colors in video log images have much broader color distribution than the MUTCD color specifications. Therefore, it is difficult to use any deterministic segmentation method to recognize the original MUTCD color class. A sophisticated model should be developed to describe the actual sign color distributions in the actual video log images so that we can evaluate the sign color in a more reliable and accurate way. In the algorithm, the Color Statistical Model (SCM), developed in our lab, is used for sign color processing. SCM is based on the specifications of the Manual Uniform of Traffic Control Device (MUTCD, 2008). SCM can successfully process the colors of sign background and legend, thereby providing reliable results for image segmentation and sign color feature analysis. SCM has good ability for general MUTCD sign color processing because it is based on the statistical colors that were collected from video log images and trained by the Artificial Neural Network (ANN) with Function Link Network (FLN) structure. The following paragraph presents the basic introduction to SCM. The SCM color model uses cases with a given input pixel value that has the probability of A to be a MUTCD color X and a probability of B to be an MUTCD color Y. The MUTCD SCM was first built statistically using labeled traffic sign color samples. The dataset for the experiment is excerpted from the LADOTD roadway video image sets. From 45,151 video log images captured under various outdoor lighting conditions in Louisiana, 3,023 images were identified as having a total of 5,052 traffic signs of 62 different types. All of the traffic signs were manually color labeled according to one of the 10 MUTCD colors. Finally, a total of 413,724 distinct samples and each reference count were used to build ground-truth probability. 2.2 Shape feature extraction Sign shape is another important feature for traffic sign detection. In this algorithm, we used the polygon approximation based algorithm for shape detection. We first identify the boundary region of a traffic sign and then analyze the features within the boundary region to determine if it is a candidate for identification as a traffic sign. Such procedures proceed from the fact that 99.4% of traffic sign types are convex, and 99.8% of those convex traffic signs have a limited number of vertices. Besides, even non-convex traffic signs (for example, the cut-out shield type) typically appear within information class traffic signs that have a rectangular boundary with a green background. As a result of such commonalities, strong assumptions can be made about traffic sign detection: (1) A traffic sign is convex, and (2) a traffic sign has a limited number of vertices. These assumptions lead to the conclusions that a traffic sign boundary becomes a polygon because a traffic sign is a twodimensional planar object and that boundary shape is also a plane figure with a limited number of vertices. The non-convex exceptions are rare. One example of such an exception is the X-shaped sign (W10-1) that occurs at rail crossings. However, a proprietary algorithm can be developed to detect such special objects and separate them from their backgrounds. The proposed shape feature extraction algorithm includes two steps. They are image preparation and binarization and nested contour chain detection for polygon approximation. 3

2.3 Sign location feature extraction The locations of traffic in the images are usually focused in several specific regions, such as the topright area, in a typical video log images. This is due to the fact that a sensor van usually goes along the roadway direction and the camera is fixed onto the van so that the locations of traffic signs demonstrate some distribution in the video log images. The main objective of sign location Probability Distribution Function (PDF) is to propose a quantities model to describe such location distribution. In such a model, a location, which corresponds to a pixel location in the image, has the probability score ranging from zero to one. High probability represents that it is more likely that traffic sign appear in such location. By introducing sign location PDF, it is useful to evaluate the sign location information. Besides, we can also use different thresholds for different applications. Figure 2 below shows a sign location distribution map, which was generated from various traffic signs in the actual video log images. We can see that the sign location demonstrates highly non-uniform distribution. Among the PDF image shown in Figure 2, it shows the sign location distributions from the manually tagged signs with a number of 1000 sign images. The darker of a location (or pixel) indicates the higher probability a traffic sign is likely to appear. It has demonstrated the phenomena of dominant non-uniform sign location distributions. In some areas, such as the bottom left and bottom right, the traffic sign would never appear. Figure 2, Sign location distribution from 1,000 images Sign location distribution is very usual for us to remove false positive in both traffic sign detection and recognition. The candidate detected in the location, where the sign location PDF is very low, such as on the roadway, at the right corner of the image, will be rejected with high probability by the algorithm. By utilizing such sign location constraints, the detection and recognition rates will be greatly improved. 3 Experiment test The section presents the experimental test on evaluating the performance of the developed sign detection algorithm. There are two basic criteria to evaluate the performance of the developed sign detection algorithm. They are false negative (FN) and false positive (FP). TP refers to true positive and TN refers to true negative. FN means that number of the image that contains sign but is detected as no-sign images. This is an indicator heavily related to system reliability. FP means that number of the non-sign image that is detected as having traffic sign. This is an indicator related to productivity (how much manual effort can be saved). The developed algorithm was tested with actual video log images provided by the City of Nashville. A total of 1,105 video log images are tested. The image resolution is 1300 * 1030. The 4

images were taken at an interval of 20 ft. (approx. 6 m) between two consecutive images. These images cover a distance of 15km. The testing site for these video log images is in an urban area, where the image backgrounds are very complicated with a lot of sign-like shapes and objects, e.g., the advertisements, the windows on the wall, and other non-traffic signs on the street. Among these images, 183 images have traffic signs, accounting for 12.3% of the total images. We used developed sign features of color, shape, location PDF, sign area, sign distortion angle, for traffic sign detection. The results are presented in Table 1 below. Table 1. Sign detection results from Nashville video log images Sect# TP TP % TN TN % FP FP % FN FN% 1 17 100 57 79.2 15 20.8 0 0 2 26 100 12 80 3 20 0 0 3 5 100 14 33.3 28 66.7 0 0 4 4 100 35 89.7 4 10.3 0 0 5 5 100 13 33.3 26 66.7 0 0 6 9 100 26 100 0 0 0 0 7 2 100 53 94.6 3 5.4 0 0 8 2 100 5 100 0 0 0 0 9 3 100 9 60 6 40 0 0 10 1 100 0 100 0 0 0 0 11 12 100 12 70.6 5 29.4 0 0 12 15 100 42 70 18 30 0 0 13 9 100 9 25 27 75 0 0 14 2 100 0 100 0 0 0 0 15 3 100 4 50 4 50 0 0 16 18 100 21 53.8 18 46.2 0 0 17 2 100 0 100 0 0 0 0 18 13 100 24 64.9 13 35.1 0 0 19 11 100 24 100 0 0 0 0 20 24 100 306 78.1 86 21.9 0 0 Total 183 100 666 72.2 256 27.8 0 0 The results show that the algorithm can achieve a zero false negative rate while keeping the false positive rate as low as 27.8%. Therefore, with the proposed algorithm, we can cut more than 72.2% of the images containing no signs that do not need manual review. These results further demonstrate that the proposed sign detection algorithm is very reliable even in the complicated environments. Based on the above discussion, if the algorithm outputs are reliable, agencies need to only review 439(256 +183) out of total 1,105 images, which is approximately 39.7%. In other words, 60.3% of the workload in manual review can be saved with the proposed algorithm even in a very complicated roadway conditions, such as on a unban street. 5

4 Conclusions and recommendations This paper presents a generalized sign detection algorithm developed to detect the more than 670 types of sign specified in the MUTCD, which is a technically challenging. Sign detection is for the purpose of eliminating the images containing no sign and keeping the images containing signs to minimizing the efforts of reviewing images for collecting signs. After studying the MUTCD sign standard, we used the features of sign color, sign shape, sign location PDF, and other sign features. We developed an SCM color model to process MUTCD color for video log images. We used the polygon detection based algorithm to analyze sign shape. We statistically studied the sign location distribution in video log images and found that it has a non-uniform distribution. These features are generalized from video log images and MUTCD standard, which provide reliable and effective ways for sign detection. The proposed algorithm has been tested with the video log image provided by the City of Nashville. The results show that the algorithm can achieve 27.8% false positive rate while keeping zero false negative rate, which means that 72.2% of the workload for manually reviewing images are saved. As a result, the algorithm could greatly help users save time and improve efficiency, which could also enhance roadway infrastructure data collection for intelligent sign inventory system. Acknowledgements The work described in this paper was supported by the National Academy of Sciences (NAS), National Cooperative Highway Research Program (NCHRP), Innovations Deserving Exploratory Analysis (IDEA) program. The author would like to thank City Metro for providing the testing data and the data processed and analyzed by Dr. Zhaozheng Hu and Chengo Ai. References BALLERINI, R., CINQUE, L., LOMBARDI, L. and MARMO, R., 2005. Rectangular traffic sign recognition. Image Analysis and Processing - Iciap 2005, Proceedings 3617, 1101-1108. DE LA ESCALERA, A., ARMINGOL, J. M. and MATA, M., 2003. Traffic sign recognition and analysis for intelligent vehicles. Image and Vision Computing 21, 247-258. MANUAL ON UNIFORM TRAFFIC CONTROL DEVICES. FHWA, U.S. Department of Transportation, 2008. TSAI, Y. and WU, J., 2002. Shape- and texture-based 1-D image processing algorithm for real-time stop sign road inventory data collection. ITS Journal (Intelligent Transportation Systems) 7, 213-234. TSAI, Y., KIM, P. and WANG Z., 2009. A Generalized Image Detection Model for Developing a Sign Inventory, ASCE Journal of Computing in Civil Engineering, Vol. 23, No. 5, pp. 266 276. WU, J. P. and TSAI, Y., 2005. Real-time speed limit sign recognition based on locally adaptive thresholding and depth-firstsearch. Photogrammetric Engineering and Remote Sensing 71, 405-414. WU, J. P. and TSAI, Y., 2006. Enhanced roadway inventory using a 2-D sign video image recognition algorithm. Computer- Aided Civil and Infrastructure Engineering 21, 369-382. YANG, H. M., LIU, C. L., LIU, K. H., and HUANG, S. M., 2003. Traffic sign recognition in disturbing environments. Foundations of Intelligent Systems 2871, 252-261. ZHU, S. D., ZHANG, Y., and LU, X. F., 2006. Detection for triangle traffic sign based on neural network. Advances in Neural Networks - Isnn 2006, Pt 3, Proceedings 3973, 40-45. 6