Road Sign Analysis Using Multisensory Data

Transcription

1 Road Sign Analysis Using Multisensory Data R.J. López-Sastre, S. Lauente-Arroyo, P. Gil-Jiménez, P. Siegmann, and S. Maldonado-Bascón University o Alcalá, Department o Signal Theory and Communications Polytechnic School, A-2 Km. 33, Alcalá de Henares - Madrid, Spain {robertoj.lopez, sergio.lauente, pedro.gil, philip.siegmann, saturnino.maldonado}@uah.es Abstract. This paper deals with the problem o estimating the ollowing road sign parameters: height, dimensions, visibility distance and partial occlusions. This work belongs to a ramework whose main applications involve road sign maintenance, driver assistance, and inventory systems. From this paper we suggest a multisensory system composed rom two cameras, a GPS receiver, and a distance measurement device, all o them installed in a car. The process consists o several steps which include road sign detection, recognition and tracking, and road signs parameters estimation. From some trigonometric properties, and a camera model, the inormation provided by the tracking subsystem and the distance measurement sensors, we estimate the road signs parameters. Results show that the described calculation methodology oers a correct estimation or all types o traic signs. 1 Introduction Several works have recently ocused on traic sign detection and recognition [1], [2], [3], [4], [5] and [6]. Speciically, in [7] it is described the ramework that we use in this research. In this paper we handle the task o automatically estimating height, dimensions, visibility distance and partial occlusions o road signs. The aim o this work is to get an eicient ramework which could be used in a inventory system, which could provide all o these estimated parameters or each sign, and not only the type. From this work, we present a complete development based on trigonometric relationships and a camera model to estimate the listed beore parameters o road signs. This paper is organized as ollows. Section II shows a global vision o our system, and in section III we describe all the implemented algorithms to achieve the traic sign parameters estimation. Sections IV and V report the obtained results and present the conclusions o this work, respectively. 2 System Overview For this work we have built a complete inventory system as it is shown in Fig. 1, which has the ollowing subsystems: J. Mira and J.R. Álvarez (Eds.): IWINAC 2007, Part II, LNCS 4528, pp , c Springer-Verlag Berlin Heidelberg 2007

2 252 R.J. López-Sastre et al. Capture subsystem. We have mounted two irewire cameras on a car. The irst one is entrusted to capture the road signs with a ramerate o 15 ps (rames per second), and with the second one we want to capture the milestones o the route. This subsystem synchronizes the capture o these two cameras, and it saves the captured images in the hard disk o a laptop. A Positioning subsystem which is composed o a GPS (Global Positioning System) receiver and a Distance Measurement Device (DMD). The aim o this block is to acquire the GPS position o the vehicle, each second, and to measure the covered distance by the car. This subsystem sends all o these measured data to the Traic sign analysis subsystem. Traic sign analysis subsystem, which will develop two task. Firstly, the systems tries to detect, recognize and track the traic signs in the images, and in a second step, the system perorms the estimation o parameters. Fig. 1. Diagram with all subsystems which are part o the complete system In Fig. 2 we show a complete low diagram with all the tasks that this system realizes. In the last step it is where weimplementtheparametersestimation process which is the main ocus o this paper. It is important to note that this estimation step is possible only ater the system has completed the identiication o a road sign, which implies our stages: segmentation, detection, recognition and tracking. Speciically, in [7] the author describes all o these steps.

3 Road Sign Analysis Using Multisensory Data 253 Fig. 2. System low diagram 3 Traic Sign Parameters Estimation There are at least 350 o dierent road sign types. As we have described in the previous section, our system allows to detect and identiy road signs in a sequence o video. This kind o systems require detailed knowledge o the sign, with the additional purpose o reducing alse alarms and to make an inventory not only with the type o the ound road sign during the route, but also with some parameters o each sign: height, dimensions, visibility distance, partial occlusion, GPS position, distance o separation between them. For an inventory system, these kind o data could be used to veriy i the dimensions are standard, i two consecutive road signs are too closer, or i it exists any kind o partial occlusion which complicates that a driver can perceive the signal. To obtain these estimations o parameters we have considered a pin-hole camera model, as shows Fig. 3, where they are known the ollowing parameters: the ocal length (), size o each unit cell (u ch and u cw, which are the height and width o a cell, respectively) in the Charge-Coupled Device (CCD) o the camera, the height o the camera (h c ), the height o the middle point o a captured image (h m ), and the distance between the camera plane and the middle point plane o a captured image d m. Then, beore starting to record a route, we have to complete a calibration step, which consists in to measure h c, h m and d m. We develop this step on a known area, where we can measure these distances without diiculties.

4 254 R.J. López-Sastre et al. Fig. 3. Calibration o the system 3.1 Height Estimation Our irst objective is to estimate the height o the sign. Ideally, we could consider that the plane where is the road sign, and the plane where is projected the image in the camera, are parallel, but it does not happen in a real situation, because thesignorthecamera,orboth,canbeinclined. We have to take into account this consideration to design a model which lets us to estimate the height o an object. Anyway, we must to deine the model where camera and sign plane are parallel, and then, we will extend this explanation to the real situation. Fig. 4. Ideal situation where camera and road sign plane are parallel Figure 4 shows the created model to estimate the height o a road sign. Ater the detection and recognition process, we have the position o the identiied

5 Road Sign Analysis Using Multisensory Data 255 road sign in the image h si = h sp u ch,whereh sp is the height in pixels and u ch is the height in mm o each pixel. The Positioning subsystem gives the covered distance between two images where the recognition subsystem had identiied the same road sign. Then, knowing these data and that h c = h m we can establish that tan (β) = h si, (1) tan (β) = h s h c d + x, (2) where x is the covered distance between two consecutive captured images where the signal had been recognized. For the second captured image we can deine tan (β )= h si, (3) tan (β )= h s h c d From both pair o equations we can write. (4) h si = h s h c d + x, (5) h si = h s h c d. (6) These last two equations orm the ollowing system o equations h s h si d = h c, (7) h s h si d = h c + h si x, (8) where the unknowns are h s and d. Our objective is to obtain the height o the road sign h s which is h s = h c + h si h six (h si h si). (9) We only have considered the situation in which the road sign is over the middle point o the captured image. O course, we have two possible situations more. The traic sign can be at the same height that the middle point, then h s = h m. The last situation is that h s <h m, and then we can write tan (β) = h si, (10) tan (β) = h c h s d + x, (11)

6 256 R.J. López-Sastre et al. tan (β )= h si, (12) Now h s is deined as ollows tan (β )= h c h s d. (13) h s = h c h si h six (h si h si). (14) In a real approach we must consider that the road sign and the camera plane are not parallel. Figure 5 shows this situation. Now, the angle α 0,andina irst stage o calibration we must measure this angle by ( ) hm h c α =arctan. (15) d m Fig. 5. Real situation where camera and road sign plane are not parallel From this point, with α>0, we can deine ( ) hsi β =arctan, (16) tan (α + β) = h s h c d + x. (17) For the next captured image we have ( ) h β =arctan si, (18)

7 Road Sign Analysis Using Multisensory Data 257 tan (α + β )= h s h c d Then, we have the ollowing system o equations. (19) h s tan (α + β) d = h c +tan(α + β) x, (20) h s tan (α + β ) d = h c. (21) From this point, ollowing the same manner o reasoning that in the ideal situation, we will have that h s must be estimated as ollows h s = h c x tan (α + β)tan(α + β ) tan (α + β) tan (α + β ), (22) Also, i we consider the situation where h s <h m,thenh s will be estimated as h s = h c x tan (α β)tan(α β ) tan (α β) tan (α β ). (23) This mathematical study, about the trigonometric relationships between the real world and the projected image in a camera, lets us to estimate the height o a sign successully. 3.2 Dimensions Estimation To estimate the dimensions o a road sign is a task which is related with the type o the sign. We must know, beore the dimensions estimation step, the answer to the ollowing question: what kind o traic sign do we have recognized?. With this inormation, we only have to apply the same algorithm that we have described in the previous section, but not only or the botton point o the sign. The detection and recognition subsystem gives us the vertexes o the rectangle where the road sign is conined. For a correct dimensions estimation, we have to estimate the height o the top and botton corner o this rectangle. Then, we can compute all the dimensions or every type o road sign. 3.3 Visibility Distance Estimation We deine the visibility distance as the distance where the road sign has been detected or the irst time by our system. This parameter is crucial or and inventory system which wants to measure the level o saety o a road. To estimate this parameter, we have to take up again the approach to estimate the height o a road sign. Figure 4 shows the model or a ideal situation where road sign and camera plane are parallel. Equations (7) and (8) compose a systems o equations, where the unknowns are d and h s. We can work out the value o d, whichisthe visibility distance that we are searching. d = h six h si h si. (24)

8 258 R.J. Lo pez-sastre et al. In a real situation we have to use the model presented in Fig. 5. Then, we can estimate the visibility distance as 3.4 d= tan (α + β) x i hs > hm tan (α + β ) tan (α + β) (25) d= tan (α β) x i hs < hm tan (α β ) tan (α β) (26) Partial Occlusion Estimation Many times, traffic signs appear occluded by other objects like trees, vehicles, other road signs, etc. This traffic signs analysis system gives an estimation o partial occlusions, and to have this inormation o each sign results crucial or a complete inventory system which measures the level o maintenance o a road, including the saety. To estimate this parameter, we use the inormation provided by the tracking subsystem: i the same road sign is detected in non-consecutive rames, we can determine that a partial occlusion was happen. 4 Results The current data o several traffic signs have been obtained or a sequence in a two ways road. For testing our approach we have measured the height and dimensions o some road signs, which are presented in Fig. 6. (a) (b) (d) (c) (e) Fig. 6. Some detected and recognized road signs

9 Road Sign Analysis Using Multisensory Data 259 Table 1. Parameters estimation or some road signs Road Sign Real Height (m) Estimated Height (m) Error (%) a) b) c) d) Our test sequences o images have been recorded with two irewire cameras ixed onto the hood and the roo o the car. The captured images have a resolution o , and in a calibration step we have measured h c, h m,, d m and u ch. Figure 6 shows in image e) a milestone detected by our system, and Table 1 shows the estimated height or each type o sign and the error commited. We can observe that the error obtained is not too high, and the parameter estimated is closer to the real value. The worse the measurement in the calibration step, the larger the error in the estimation. The most important cause o error in the estimation process is the measurement o the variable x, which indicates the covered distance between two captured images. The Positioning subsystem rereshes this variable each 1.6 meters, because this is the resolution o our DMD, then we do not have the exact value o this parameter. The presented mathematical procedure works well when all the variables are measured correctly, and the error obtained in this situation is about 0.8%. But the error in this estimation can be caused by some other actors: lens distortion, an incorrect detection o the sign in the image. The dimensions o each sign are estimated ollowing the same process, but or two point o the signal, as we have described in the previous sections. Table 2 shows some estimated visibility distances. Table 2. Visibility Distance Estimation Road Sign Visibility Distance (m) a) 41.2 b) 37.6 c) 42.7 d) Conclusions This paper describes a complete ramework to estimate height, dimensions, partial occlusions and visibility distance o traic signs. The integration o the recognition, tracking and positioning subsystems, allows to estimate these parameters automatically, ollowing the mathematical approach presented rom this work.

10 260 R.J. López-Sastre et al. Future lines o work will ocus on estimating more attributes o each sign, such as deormation level, distance rom the sign to the margin o the road, level o slope. On the other hand, we will work in a more complete system which will have calibrated cameras and a Positioning subsystems which improves the estimation o covered distance. Another uture objective will be to complete the system with an stereo vision ramework, in order to improve these estimations o parameters. Acknowledgment This work was supported be the project o the Ministerio de Educación y Ciencia o Spain number TEC2004/03511/TCM. Reerences 1. Maldonado, S., Lauente, S., Gil, P., Gómez, H., López, F.: Road-sign detection and recognition based on support vector machines. IEEE Trans. on Intelligent Transportation Systems (2006) 2. de la Escalera, A., Armingol, J.M., Pastor, J.M., Rodríguez, F.J.: Visual sign inormation extraction and identiication by deormable models or intelligent vehicles. IEEE Trans. on Intelligent Transportation Systems 15 (2004) Fang, C., Chen, S.: Road sign detection and tracking. IEEE Trans. on Vehicular Technology 52 (2003) Lauente Arroyo, S., García Díaz, P., Acevedo Rodríguez, F., Gil Jiménez, P., Maldonado Bascón, S.: Traic signs classiication invariant to rotations using support vector machines. In: Proceedings o Advabced Concepts or Intelligent Vision Systems, Brussels, Belgium (2004) 5. de la Escalera, A., Moreno, L.E., Salichs, M.A., Armingol, J.M.: Road traic sign detection and classiication. IEEE Trans. on Industrial electronics 44 (1997) Aoyagui, Y., Asakura: A study on traic sign recognition in scene image using genetic alhorithms and neural networks. In: Proceedings IEEE Int. Con. Industrial Electronics, Control and Instrumentation. Volume 3., Taipei, Taiwan (1996) Lauente-Arroyo, S., Maldonado-Bascón, S., Gil-Jiménez, P., Gómez-Moremo, H., López-Ferreras, F.: Road sign tracking with a predictive ilter solution. In: Proc. o IEEE IECON, Paris, France (2006)