特集 Road Border Recognition Using FIR Images and LIDAR Signal Processing

Transcription

1 デンソーテクニカルレビュー Vol 特集 Road Border Reognition Using FIR Images and LIDAR Signal Proessing 高木聖和 バーゼル ファルディ Kiyokazu TAKAGI Basel Fardi ヘンドリック ヴァイゲル Hendrik Weigel ゲルド ヴァニーリック Gerd Wanielik This paper addresses the detetion and traking of road borders in non - ooperative environments. A 2-dimensionalsanning LIDAR is used to improve the reliability of the FIR amera based road border reognition. In order to detet the road boarders we apply a Kalman-Àlter based model Àtting strategy. Extrated measurements of the FIR images are transformed into the vehile oordinate system in order to provide a preise desription of the road ourse ahead. The model desription ina ommon oordinate system - the vehile oordinate systemallows an easy ompensation of the ego-motion and a diret and straight forward fusion of the different sensor data. Both the detetion and the estimation were developed and enhaned for the intended sensor onàguration. The orresponding mathematial derivations are presented in this paper. Using the range values, delivered by the LIDAR, a more stable estimation of the pith angle an be ahieved. The realization of that is shown in detail. This is used to deàne the ROI in whih the image proessing is arried out. the refleting lane markings. The referene6) uses a one layer １. INTRODUCTION Investigations about lane road detetion systems resp. LIDAR for the detetion and traking of the road borders Have already been arried out for more than two deades without the need of the existene of lane markers. LIDARs are 1) 4) ). Mostly, a video vision based system similar unattahed to different light onditions like FIR ameras. detets and traks the road markings and/or boundaries. Moreover, as an ative sensor the LIDAR measures distanes Suh approahes have impressive results in ooperative diretly and therefore more preisely than vision based sensors. (the referenes areas under good weather onditions. In ontrast to that FIR sensors turn out their strength and benefits mainly under ２. THE COMBINED SENSOR SYSTEM noturnal onditions within nonooperative environments. Taking the advantages of both sensors into aount a In omparison with video sensors the high tolerane towards ombination ould lead to an improved and more reliable different light onditions makes a FIR sensor also suitable for system for road border traking. As mentioned above eah senarios with extremly dif ult light onditions like shadow sensor as stand-alone system is able to ful ll a desired DAS or bak-light. funtionality. So, a road boarder detetion would not lead to FIR ameras are reently used for Advaned Driver Assistane Systems (ADAS) in passenger ars. HONDA additional hardware osts. Already existing algorithms like objet reognition9) will also be used. and BMW optionally prepare a FIR amera system for Figure 1 gives an overview of the road border traking their flagship ar and espeially the HONDA system also system. A stand-alone approah using the FIR amera will reognizes pedestrians. LIDAR sensors are installed instead be desribed in the next setion. The LIDAR improves the of a radar for ACC funtionality or as a pre-rash system. basi system. It sustains the road boarder reognition within TOYOTA and NISSAN have been using the LIDAR for their the overall system in different ways. The LIDAR emits the ADAS suh as ACC. SUBARU employs the LIDAR for all- laser beam forward with 2-dimensional sanning by the speed ACC on their vehile in rotating hexagon mirror to detet the forward objets of the Although not indented for a ommerial appliation so far egovehile. The mass-produed LIDAR only uses the upper LIDAR systems are also used for lane reognition purposes. 3 mirrors to detet ahead vehiles for ACC system but in In the referene5) a 2-dimensional sanning LIDAR detets our system we also use the lower 3 mirrors (see Fig. 2). As Reprinted with permission from Intelligent Vehiles Symposium, 2007 IEEE Digital Objet Identifier: /IVS Publiation Year: 2007, Page(s):

2 shown in these speifiations, exept the elevation eld of view, are the same as the mass-prodution type. The LIDAR signal proessing supports the FIR proessing at different points: First, with the output of a objet traking algorithm we exlude regions from the ROI whih are overlayed by other objets 9). With the lower 3 layers a pith angle estimation is done to improve the transformation between vehile oordinate system and image plane. Depending on ego-vehile position on the road a ombined range and refletivity signal proessing distinguishes if the sanned area is on or beside the road. A detailed explanation of the reetivity signal proessing and its usage onerning the detetion of the road area an be found in the referene 6). FIR LIDAR Image Proessing Range Refletion ROI objet traking 3 upper layer 3 lower layer 3 upper layer 3 lower layer Trans form Adaption Data Fusion Fig. 1 Sheme of the overall system Upper 3 Layer (4.4 degree) Upper 3 Layer (4.4 degree) Fig. 2 Detetion angle (elevation angle) Item Range Azimuth Field of View Elevation Field of View Caluulation frequeny Horizontal Resolution Size Speifiation 120 m deg deg 10 Hz 0.08 deg. W100*H60*D80 Fig. 3 Speifi ations of the LIDAR In this setion a omplete approah for solving the problem of the road border detetion in FIR-images is presented. The way, how to inrease the robustness of this approah with the help of the multi layer LIDAR will be disussed in the next setion. Before going more into detail, it is useful to introdue the most important properties of the FIR-imaging regarding the intended appliation. The textural differene between the road surfae and the surrounding area in FIR-images is usually very poor. The road border an mostly be reognized as some disontinuities in the FIR-image. The FIR-images are sometimes very noisy depending on the weather ondition. Sine suh noise appears as variations of the image struture, it an ause some difulties during the detetion proess. Further objets (pedestrians, trees, rash barriers et.), whih our near to the road border, an also ause some diffiulties beause of the orresponding gray-sale variations. Taking these items into aount we extended our road border detetion algorithms introdued in the referene 6) and 7) at different points. In both systems we followed a model tting strategy, in whih the road border is approximated by means of a geometri urve. Suh approah shows robustness against noise and interruptions. The proedure of the assoiated proessing is typially subdivided into two steps: measurements extration and model parameter estimation. Using the Kalman-Filter approah and the vehile oordinate system in order to desribe the road urve, we are able to ompensate the ego motion and to give an exat estimation of the road border. The way how to onsider the ego motion in the estimation proess is one of the enhanements presented in this paper. With it the proessing is signiantly simplied without restriting the suitability of the method. The seond point of our enhanements is the usage of the loal orientation in the image plane in order to inrease the reliability of the measurements. Therewith a lot of irrelevant measurements aused by other objets an easily be exluded. With the help of the struture tensor method the loal orientation an be measured on the one hand and on the other hand we are able to distinguish between gray-sale variations aused by noise and disontinuities orresponding to relevant image strutures. The use of suh a method in the extration of

3 measurements is also one of the important extensions of our algorithms. A. Model parameter estimation As mentioned above a Kalman-Filter approah is used to estimate the parameters of a predetermined urve desribing the road border. Beause of the intended data fusion of different sensors, measuring and estimating domains are ompletely separated. While the vehile oordinate system v x denes a ommon domain, in whih the urve parameters are estimated, the measuring domains are different depending on the measuring priniple of the sensors. In the ase of the FIR amera the measuring domain is the image plane i x and the transition into the estimating domain is non-linear. We denote this transition f and the transition between two time steps f a. To estimate the urve of the road on the basis of the measurements of the FIR-amera and the LIDAR, a funtional model is onsidered. A urve with onstant urvature, a irle, is hosen. It suits for the typial distanes, whih an be observed by the LIDAR in our appliation as well as for the set of distanes, for whih the position of the road border is measured in the FIR-amera image. The original model equation is transformed into a form suitable for the estimation purposes. It is adapted with the help of a shift in the y-axis (y 0 ) and a orretion term of the urvature parameter representing the line position of the left (i=1) or right (i=0) road boarder: The sign of the radius 1 respets the diretion of the urve. In the simple ase we an aept that the vehile moves along the irle segment with the radius and the deviation from the initial orientation of the vehile is modelled as noise of the state parameters. The state vetor in this ase onsists of the three parameters desribing our irle model:, b and y 0. To get some preditions in order to perform the update step of the Kalman-Filter, a set of distanes v x n is determined and put in the last equation. The orresponding v y n -oordinates an be omputed if the equation is onverted into a form where v y is a funtion of the hosen distane v x as well as of the parameters ontained in the state vetor: Finally, the orret solutions and the orresponding v x n are transformed into the image plane to obtain a new estimation of the model parameters. However, this assumption leads to an inaurate tting if the orientation of the vehile deviates from the assumed orientation signifiantly. Therefore we make this quantity deterministi and estimate it with the model parameters. The state vetor turns out to be: where y v (t k ) is the offset parameter indiating the shift in the y-axis. All of the parameters are desribed now in a irle oordinate system, whih is introdued to desribe the relative position of the vehile with regard to the irle (). To simplify the proessing we orret the relative position of the oordinate systems x and v x by onsidering the ego motion 1 at eah time step, when the estimation is performed. An additional offset x v should be avoided. In the first step the predition step of the Kalman-Filter is arried out in the urrent irle oordinate system, where the state (tk-1) x v (t k-1 ) = 0, y v = (tk-1) y v (t k-1 ) and v = (tk-1) v (t k-1 ) is taken as a starting point. Using the atual veloity (tk-1) y v (t k-1 ) and turn rate (tk-1) v (t k-1 ) of the vehile we obtain for the next time t k and the time interval T the following interim state: with The movement of the vehile is aompanied by the movement of the tangential irle oordinate system afterwards. Lastly, the new position of the vehile has to be transformed into the new irle oordinate system and has to be expressed again with the state spae variables desribed above. We assume that both the radius of the irle and the width of the road remain unhanged during the predition step: 1 The ego motion is measured by means of additional sensors.

4 (t k ) y (t k-1 ) y The new offset parameter is alulated using the position parameters (tk-1) x v (t k ) and (tk-1) y v (t k ): (t k-1 ) γ (t k-1 ) (t k ) (t k-1) y (t k ) y (tk-1) γ (t k ) γ (t k ) And the new angle parameter is delivered by (tk-1) y (t k-1 ) (t k-1 ) y (t k ) (tk-1) x (t k ) y (t k ) (t (t k-1 ) k-1 ) x (t x (t k-1 ) k ) (t k) x (tk) x (t k-1 ) x This two equations together with the parameters (t k ) and b (t k ) are the elements of the vetor f a. The way to dene the vetor f is disussed in the next subsetion. B. Extration of measurements The extration of measurements from the FIR-images is normally ahieved by deteting the signifiant graysale variations, whih our around the predited urve. As mentioned in the last setion in our system we onsider further properties of the image struture in order to inrease the reliability of the estimation. On the one hand we distinguish between noise and relevant graysale variations, on the other hand the orientation of the loal image struture is additionally taken into aount. Using the struture tensor approah 8) both properties an be deteted. The struture tensor is a symmetri 2 2 matrix J, and its omponents are dened as follows: g (x') g (x') The vetor x ontains the oordinates of a pixel in the image g (x), and is the window funtion whih determines the size and shape of the neighborhood around the pixel. Through a rotation of the oordinate system, the struture tensor is brought into a diagonal form with eigenvalues J 11 and J 22. The orientation is given by the rotation angle and omputed as Fig. 4 Funtional road model and the vehile predition steps The lassifiation of the image pixels as homogeneous, isotropi or oriented is performed by the interpretation of the eigenvalues as follows: Homogeneous, J 11 = J 22 = 0: where the threshold Ch is determined by the noise level in the image Isotropi, J 11 = J 22 0: Oriented, J 11 = J 22 = 0: To reate uniformly distributed preditions in the image plane we define the v x -oordinates with the help of a nonlinear geometri series: where v x min is an offset aording to the displaement of the amera in the vehile oordinate system. This parameter and the initial distane v x 0 as well as k are determined

5 empirially. The orresponding v y-oordinates an be dened using the model equation Its transformation into the amera oordinate system an be performed in onsideration of the amera rotation: where v x m and v y m are the enter oordinates of the irles in the vehile oordinate system: and the orresponding auxiliary point turns out to be: Now we an supply the last omponent of f by transforming both of the points into the image plane and by giving the slope dened between them: The v y -oordinates are then obtained by onverting the equation 14: where the sign of the root depends on the sign of the urvature. The related omponents of the f -vetor are then given by the perspetive projetion equations: where N x N y is the number of the pixels, f the foal length, (ddx;ddy) the sensor dimension and is the transformation into the amera oordinate system with the help of the rotation matries and the translation vetor desribing the amera position in the vehile oordinate system. To define the omponent of f regarding the loal orientation we introdue a set of auxiliary points, whih is reated by means of the tangential vetors at the points ( v x (i;n); v y (i;n)) (). The tangential vetor is defined as follows: The analysis of the FIR images regarding road border detetion in non ooperative areas has shown that for an effiient extration of relevant features it is neessary to restrit the proessing on small areas lose to the road borders. These regions of interest are determined using the urve parameters whih are estimated in the last fusion step. This is illustrated in with the feedbak-branh. For an aurate plaement of the regions of interest in the image, the transformation parameters between vehile oordinate system and image plane must be adapted for eah time step. This is performed with the help of the pith angle whih is determined using the range measurements of the laser sanner: In front of the host vehile the 3 lower layers hit the road surfae in a distane between 5 and 12 meters. For a dediated lateral span (in aordane with the vehile width of 1.7 meters) a line for eah layer is alulated. This is also done during the initial alibration proess. A displaement of these lines is related to the hange of the pith angle. shows an example of the distane measurement of the rst and seond layer. Depending on the urrent pith angle not all lower layers an be used always. As the LIDAR is attahed on the right side of the front bumper the used distane

6 x is the distane measurement whih results from the range detetion in the initial position. The rotation angle of the sensor at the y axis in the initial state is alulated by: 1 dy d x y 1 -b d x x x m with the help of the vetor r, the angle and the measured distane l delivered by the range detetion, the urrent pith angle an be alulated as follows: y y m y y γ measurements (with lines onneted points) lie between -0.4 and 1.3 meters. The points luster on the right indiates the road edge. For eah layer the line is omputed and plotted in with the orresponding olor. Loal irregularities on the road surfae ould bastardize this omputation. To avoid suh mistakes an overall line (bold blak one) is alulated and applied for the pith angle ompensation. Furthermore the updated transformation matrix is used to alulate the orret referene point in vehile oordinates for every road border detetion in the image plane. The method how to determine the pith angle is explained below. The illustrates the position S of a mounted sensor in the vehile as well as the geometri measures at the initial position of the vehile. The origin of the vehile oordinate system is loated below the rear axle. The point D represents an arbitrary dened rotation axis, whih must be unhanged. The length of the vetor r and the angle are also onstant and they are alulated by the values of the initial position (denoted by the index 0): where Fig. 5 Determination of the tangential vetor and And so the pith angle an be used to alulate the atual translation vetor r s and the urrent rotation matrix R y : Consequently, the new relative position of the sensor an be ompletely desribed and the transformation of a point from the vehile oordinate system into the amera oordinate system turns out to be: The introdued approah was tested with seleted offline senarios and under real environment onditions. It turned out that the LIDAR based pith angle ompensation is an appropriate solution to deal with the often very warped road surfaes in non ooperative environment. presents some examples of proessed FIR images and the orresponding traking results. The lower row presents the result of the detetion step (8), where the olor denotes the orientation of the grey sale variation (9). The omputation of the omplete image is just done for visualization purposes.

7 (m) (m) Fig. 6 Example of the distane of the 1st and 2nd layers measured by the LIDAR υ z r S r D D r γ δ S Fig. 7 Geometri relations of the sensor and vehile oordinates Atually, the algorithm omputes only the image region under the horizon to save omputation resoures. The upper row shows the original images overlayed with the measurements for the update step of the Kalman-Filter. As an be seen, the usage of the orientation provides a reliable way to pik out image points belonging to the road border. Consequently the ahieved estimation of the model parameters resp. the road borders is very preise. s l α β υ x 1) E.D. Dikmanns, B.D. Mysliwetz, Reursive 3D road and relative ego-state reognition IEEE Trans. Pattern Analysis and Mahine Intelligene, Vol. 14, No.2, Feb. 1992, pp ) R. Wang, Y. Xu, Libin, Y. Zhao, A Vision Based Road Edge Detetion Algorithm, Pro. IEEE Internatonal Conferene on Intelligent Vehiles, Versailles, ) N. Apostoloff, A. Zelinski, Robust vision based Lane Traking using Multiple Cues and Partile Filtering, Pro. IEEE Internatonal Conferene on Intelligent Vehiles, pp , ) R. Chapuis, R. Aufrere, F. Chausse, Aurate Road Following and Reonstrution by Computer Vision IEEE Trans. Intelligent Transportation Systems, vol. 3, no.4, pp , Deember ) T. Ogawa, K. Takagi, Lane Reognition Using On-vehile LIDAR Pro. IEEE Intelligent Vehiles Symposium, ) B. Fardi, U. Sheunert, H.Cramer, G. Wanielik, Multi Modal Detetion and Parameter-based Traking of Road Borders with a Laser Sanner Pro. IEEE Intelligent Vehiles Symposium, ) B. Fardi, G.Wanielik, Hough Transformation Based Approah for Road Border Detetion in Infrared Images Pro. IEEE Intelligent Vehiles Symposium, ) B. J ahne, Digital Image Proessing. Conepts, Algorithms, and Sientifi Appliations, Springer- Verlag, Berlin, Heidelberg, New York, ) K. Takagi, K. Morikawa, T. Ogawa, M. Saburi, Road Environment Reognition Using On-vehile LIDAR Pro. IEEE Intelligent Vehiles Symposium, Fig. 8 Examples of the proessed FIR images and the orresponding traking results

8