A parallel system for real time traffic sign recognition

Transcription

1 A parallel system for real time traffic sign recognition Volker Rehrmann Raimund Lakmann Lutz Priese Image Recognition Laboratory University of Koblenz Landau Rheinau 1 D Koblenz, FR Germany lb@uni koblenz.de Abstract We present a system for the real time recognition of traffic signs from a moving car on European highways. The traffic sign recognition system (TSR) was developed within the European PROMETHEUS project in cooperation with Daimler Benz and is installed in an autonomous car. Our TSR is also intended to serve as a driver assistance tool. The TSR is based on a fast color image analysis. This analysis involves different methods, such as an inherently parallel color image segmentation, a data driven decision graph with fuzzy techniques, and classical pattern recognition. Due to the good quality of the color segmentation and the fault tolerant evaluation the system is highly robust against the difficult conditions in natural outdoor scenes. To meet real time constraints the TSR has been implemented on a high speed parallel image processing system with MPC 601 processors. A prototype of the TSR runs in a usual car and reaches a recognition rate of 98 %. Keywords: Parallel Computer Vision, Traffic Sign Recognition, Color Image Analysis. 1. Introduction In today s road traffic lots of information has to be handled to move a car safely through traffic. Visual information is the most important kind of information in traffic scenes. A long range objective of many researchers has been the development of autonomous vehicles ([5]). An intelligent system for autonomous driving has to fulfill various tasks, e.g. road tracking, obstacle detection, traffic sign recognition, etc. An autonomous car has been one aim of the European research project PROMETHEUS (PROgraMme for a European Traffic with Highest Efficiency and Unprecedented Safety). Within this project Daimler Benz in cooperation with some universities developed the autonomous car VITA II ([12]). VITA II is able to drive autonomously at a speed of 130 km/h on European highways. The task of our group within the VITA II project was the recognition of traffic signs. Traffic sign recognition (TSR) is a recent topic in vision based vehicle guidance, see for example [1], [2], [3], [11]. Besides the integration into an autonomous car our TSR module can also be used independently as a driver assistance (e.g. by warning the driver when exceeding speed limits). Our system meets real time constraints by using parallel processing techniques and achieves high recognition rates even under difficult conditions in natural outdoor scenes. Our TSR is based on the analysis of color images. In the following chapter 2 we describe the method for recognizing traffic signs. Chapter 3 presents different approaches for the parallelization of our system. The results of our TSR have been tested with large image series and are presented in detail in chapter TSR Method One of the most important components of an image analysis system is the image segmentation. The aim of color image segmentation is a partition of an image into visually distinct regions that are homogeneous with respect to their color. The first part of our TSR system consists of a fast, robust color segmentation. The second part is the extraction of visible traffic signs based on the segmentation results. Both phases of our TSR system are described in the following sections.

2 2.1 Color Segmentation The segmentation of color images is a computationally expensive task. Most of the known techniques use sequential algorithms with a complexity far away from real time application ([7]). To allow for real time application we developed the Color Structure Code (CSC). The CSC is an inherent parallel color segmentation technique that also operates on distributed data. The CSC follows a hierarchical region growing on a special hexagonal topology (that was firstly introduced by Hartmann, [6]). This hierarchical topology is formed by so called islands of different levels. One island of level 0 consists of seven neighbored pixels in the hexagonal topology. One island of level n+1 consists of seven overlapping islands of level n (s. Fig. 1). Repeating this until one island covers the whole image, the number of islands decreases from level to level by a factor 4. island of level 2 island of level 1 island of level 0 pixel Figure 1: The hexagonal, hierarchical island structure. The CSC works essentially in four phases: In the preprocessing phase noise suppression is accomplished by the use of a nonlinear filter which, in addition, strengthens the sharpness of contours ([8]). In an initialization phase color homogeneous regions in islands of seven pixels are detected and mapped to initial code elements (s. Fig. 2(a)). Such an initial code element consists of those pixels of islands of level 0 that are neighbored and whose color distance lies below a certain threshold. Hence, a code element of level 0 describes a small colored region within an island of level 0. At most three color regions can be detected within one island of level 0. In the linking phase code elements of level n are linked to new code elements of level n+1 in seven neighbored, overlapping islands of the hexagonal island structure (s. Fig. 2(b)). Code elements will be linked if the regions represented by them are connected and similar in color. The connectivity of code elements can easily be determined within the hexagonal island structure. Two code elements are connected if they share a common subregion in their overlapping subisland. In the example of level 1 in Figure 2(b) this simply means that they share a common pixel. By repeated linking those code elements form a code tree. Code elements that do not find any partner for linking in some level n form the root of such a code tree. Thus, a connected, homogeneous region is represented by a tree in our CSC data structure. The larger a region is, the higher is its root level in the hierarchical data structure. The root contains raw information about the size, location, and mean color of a region. More details can be obtained descending the tree. A typical error in local region growing techniques is the linking of differently colored regions due to a chain of connected pixels with smoothly changing colors. Often it refers to an outflow of a region, that can t be detected locally. In the splitting phase those regions are separated, that were locally homogeneous and connected but are not homogeneous at a more global sight. We recognize these chaining mismatches by the repeated hierarchical linking of the code elements. Two code elements in some level n have to be split if they are not similar in color, although they are connected by a chain of similar colored code elements in level n-1.

3 (a) (b) Figure 2: The initialization (a) and the linkage (b) of code elements. The color similarity measure is of particular importance for the quality of the segmentation results. We achieved the best results with a newly developed measure in the HSV color space ([4]). This color space leads to very robust segmentation results due to the stability of the hue value under varying lighting conditions. More details on the CSC can be found in [10]. 2.2 Evaluation The evaluation phase of our TSR system is divided into two phases: detection and identification. Detection includes locating all signs and determining their class, e.g. prohibition sign (red circle), danger sign (red triangle), mandatory sign (blue circle), etc ([9]). Identification is realized by recognizing the various ideograms in traffic signs which are used to present the explicit information about the kind of prohibition, danger, etc. Different techniques have to be used for these two phases. (a) (b) (c) (d) Figure 3: The prohibition sign (a) is detected by the recognition of the red circle (b), the ideogram is extracted (c) and mapped to a fixed sized bitmap (d). Detection is achieved by a fault tolerant knowledge based hierarchical decision graph consisting of many different feature extraction modules. In this graph a decision is not made in a binary (yes, no) but in a fuzzy way (may be) by using probability measures. Thus, it is possible that at a certain stage several interpretations may be true leading to a concurrent processing in the decision graph. This, of course, implies the possibility of rejecting previously made decisions. The main features in the detection graph are color, shape, and topological relations. Therefore, we collect all regions of typical traffic sign colors (red, blue, yellow, white) from the CSC database into a so called candidate list. The ranges of our color classes are well defined for various lighting conditions caused by shadows or varying color temperature of daylight. This is not a trivial task as especially the color white appears in very different brightnesses. We regard the candidate list as the result of a first filter in a cascade of filters reducing the number of traffic sign candidates to the real existing traffic signs in the image. A perfect segmentation in the sense that all segmented regions correspond exactly to the real scene is not possible. Traffic signs are often split into pieces. To be robust against such situations small neighbored regions of similar color are collected into a list

4 of partial objects. Some fast tools recombine those partial objects to virtual objects and add them to the candidate list. The next step is a geometric classification of the objects from the candidate list. The shape of an object is encoded by a fast approximation of its convex hull leading to relative simple shape recognition methods. For every object a probability is calculated that it belongs to a certain shape class. These shape classes are circles, semi-circles, triangles, and rectangles depending on the color of the object. Note, that one object may belong to several shape classes which leads to the concurrent processing in the decision graph. With this utilization of color and shape we are able to detect nearly all traffic signs. However, such an analysis of color and shape alone is not sufficient. Thus, we have to verify some further features of traffic signs. These features are mainly inclusions of particular color segments and histograms of certain colors inside specific templates. The first part of the identification phase is the isolation of the ideogram from the detected traffic sign and the reduction to a binary (black, white) or ternary (black, white, red) image. This preprocessing consists of a color classification for all pixels inside a found traffic sign candidate and the elimination of the irrelevant traffic sign border. The isolated ideogram is presented by a fixed sized bitmap that becomes the input for various identification tools. Some of these tools are neural nets and nearest neighbor classifiers. They are used competitively by combining their probability results to increase the robustness of the identification. Figure 3 illustrates the principal procedure of our TSR system. 3. Parallelization There is a clear difference in the parallelization of our TSR system between the generation of the CSC and the evaluation of the CSC. It is quite easy to parallelize the CSC as it was especially designed as a parallel algorithm. It becomes more difficult parallelizing the evaluation of the CSC which is controlled by a decision graph with many sequential dependencies. In the following we will describe two different approaches to the parallelization of the TSR system. In all phases the CSC generation operates locally in one island. Both, the preprocessing and the initialization phase operate on a seven pixel island and can be processed independently of all other islands. The linking process operates independently on each group of seven islands in higher levels and can as well be done in parallel. The only sequential constraint is due to the linking hierarchy. Regions in a group of seven islands of level n can not be linked to level n+1 before all these regions have been linked to level n. Consider the CSC generation as a network of parallel processes. Every island in the hierarchy represents a process and is connected to other islands by channels following the island hierarchy. A process can start its computation as soon as the data on all its input channels have been arrived. Realized in this way the CSC generation acts as a data driven, asynchronous parallel network (s. Fig. 4). This concept shows the high potential of parallelism in the CSC generation. level n+1 level n level n-1 Figure 4: The CSC generation as a network of parallel processes.

5 However, due to efficiency reasons the implementation on a parallel system with a relatively small number of processors looks different to this logical concept. The mapping of this concept into a parallel system can be done in different ways. An obvious idea is to divide an image into equally sized frames corresponding to the available number of processors. We have implemented two different approaches following this idea. In the first approach each processor has its own database where it manages all data and processes all islands within its subimage. The image data is distributed to all processors via a fast bus. The preprocessing and the initialization phases operate directly on pixels and can be processed parallel in all frames without requiring any communication. In the linking phase processes at the borders of the frames need data from neighbored processors. Note, that only data of islands immediately located at the border of each subimage have to be transmitted (s. Fig. 5). The database of the CSC is distributed among all processors. There exists no central control process, every processor asynchronously processes its subimage and communicates with other processors. This approach is well suited for a massively parallel approach as the amount of data which has to be transferred in the linking phase is quite small. However, from an engineering point of view the parallelization of the traffic sign evaluation is more expensive in such a distributed database. Border between two subimages Figure 5: A border between two subimages. Islands of one group of islands belong to different subimages. In a second approach the complete CSC database is stored in every processor s local memory. The preprocessing and the initialization phase operate in exactly the same manner as in the first approach, but the linking phase is different. After computing each level every processor broadcasts its own processed data to all other processors. Consequently, every linking process finds the CSC data from level n needed for computation of level n+1 in its own local memory. As an advantage at the end of the CSC generation every processor holds the whole CSC database in its own memory. This simplifies the parallelization of the evaluation. All the traffic sign candidates in the candidate list can be processed independently of each other. To parallelize the evaluation phase we use the concept of a processor farm which automatically leads to an excellent load balancing. Independent sub-tasks (here: the analysis of an object of a typical traffic sign color) are farmed out by a control process to idle processors in the net. As soon as one processor has finished a sub task it sends the results to the controller and gets a new sub-task. This concept is well suited for processor nets with a relatively small number ( ) of processors and small sized data packages for the sub tasks. Both prerequisites meet in the case of our TSR application. As every processor holds the whole CSC database and the whole image in its local memory, only few communications are required for the parallel processing of the evaluation phase. The parallel hardware we use is the Parsytec TIP system (Transputer Image Processing) which was especially developed for fast image processing applications. The processors are connected by links and an additional fast video bus for broadcasting the images to all processors (s. Fig. 6). The first processor in this system is a color frame grabber as an interface to the camera. The processing units consist of a Motorola PowerPC processor (MPC 601) for the computation and a Transputer (T805) for the communication coupled by dual ported RAM. At the moment our system is equipped with four processing units.

6 camera CFG Display TIP-bus CGD PU PU PU PU Figure 6: The TIP system with a color frame grabber (CFG), four processing units (PU) and a color graphics display (CGD). 4. Results In order to test our TSR system we built a large traffic scene database of series of approximately images of them have been manually examined in order to register all visible traffic signs in protocol files. All traffic signs in these image sequences are labeled by an identification number and some attributes, e.g. position, size, visibility, etc. We use this database to compare the recognition results of the TSR automatically with existing traffic signs. During the development of the different TSR modules these statistics have delivered detailed information about the recognition rate and the number of false alarms. This greatly helped optimizing the TSR system step by step. Above a size of pixels the TSR system nearly detects all traffic signs, e.g. 98 % of prohibition signs (s. Fig. 7(a)). This size of a traffic sign is corresponding to a distance of approximately 50 meters to the camera. 100 % prohibition signs >34 size PowerPC MPC 601 segmentation 783 (86%) evaluation 129 (14%) 912 (100%) (a) (b) Figure 7: (a) Statistic of prohibition sign detection. The false alarm rate is per image. (b) Runtimes (in msec) of the TSR. Images of the size pixels were analyzed by one parallel processor PowerPC MPC 601 in a TIP system A basic requirement for a practical TSR system is the reliable recognition of all traffic signs in real time. We succeeded in developing a practical TSR system which combines high quality and a fast runtime. A high quality was achieved by a robust color segmentation and a fault tolerant, knowledge based evaluation of traffic signs. A fast runtime was reached by developing fast parallel algorithms for segmentation and evaluation. A fast processor (MPC 601) performs a cycle time of 912 msec to evaluate a pixel image (s. Fig. 7(b)). As traffic signs usually appear only in certain regions of interest (ROI) in the image,

7 we analyze only ROIs with a size of pixels. By a simultaneous evaluation of several ROIs on different processors our system is able to analyze three images per second. To increase the safety of the system we are intending to include tracking of traffic signs in a sequence of images. With our approach of a parallel segmentation and evaluation we expect to reduce the runtime to less than 150 msec with six processors. A car at a speed of 130 km/h moves only 6 meters in 150 msec. Thus, traffic signs change quite smoothly and a tracking becomes reasonable. In the context of traffic sign recognition 150 msec can be regarded as real time considering the response time of a human driver. References [1] M. Campani, M. Straforini, M. Cappello, E. Reggi, G. Piccioli, and V. Torre. Visual routines for outdoor navigation. In Proceedings of the Intelligent Vehicles Symposium, pages IEEE, Tokyo, July [2] M. de Saint Blancard. Road sign recognition: A study of vision-based decision making for road enviroment recognition. In I. Masaki, editor, Vision based Vehicle Guidance, pages Springer Verlag, New York, Berlin, Heidelberg, [3] S. Estable, J. Schick, F. Stein, R. Ott, R. Janssen, W. Ritter, and Y.J. Zheng. Real time traffic sign recognition system. In Proceedings of the Intelligent Vehicles Symposium. IEEE, Paris, Oct [4] J. D. Foley, A. van Dam, S. K. Feiner, and J. F. Hughes. Computer Graphics: principles and practice. Addison Wesley Publishing Company, second edition, [5] V. Graefe and K.-D. Kuhnert. Vision based autonomous road vehicles. In I. Masaki, editor, Vision based Vehicle Guidance, pages Springer Verlag, New York, Berlin, Heidelberg, [6] G. Hartmann. Recognition of Hierarchically Encoded Images by Technical and Biological Systems. Biological Cybernetics, 57:73 84, [7] Y. Ohta, T. Kanade, and T. Sakai. Color information for region segmentation. Computer Graphics and Image Processing, 13: , [8] M. Pietikäinen and D. Harwood. Segmentation of color images using edge-preserving filters. In V. Cappellini and R. Marconi, editors, Advances in Image Processing and Pattern Recognition, pages North-Holland, [9] L. Priese, J. Klieber, R. Lakmann, V. Rehrmann, and R. Schian. New Results on Traffic Sign Recognition. In Proceedings of the Intelligent Vehicles Symposium. IEEE, Paris, Oct [10] V. Rehrmann. Stabile, echtzeitfähige Farbbildauswertung. Verlag Fölbach, Koblenz, [11] P. Seitz, G.K. Lang, B. Gilliard, and J.C. Pandazis. The robust recognition of traffic signs from a moving car. In B. Radig, editor, Mustererkennung 1991, pages Springer Verlag, [12] B. Ulmer. VITA II Active Collision Avoidance in Real Traffic. In Proceedings of the Intelligent Vehicles Symposium. IEEE, Paris, Oct