Spatio temporal Segmentation using Laserscanner and Video Sequences

Transcription

1 Spatio temporal Segmentation using Laserscanner and Video Sequences Nico Kaempchen, Markus Zocholl and Klaus C.J. Dietmayer Department of Measurement, Control and Microtechnology University of Ulm, D Ulm, Germany nico.kaempchen@e-technik.uni-ulm.de Abstract. Reliable object detection and segmentation is crucial for active safety driver assistance applications. In urban areas where the object density is high, a segmentation based on a spatial criterion often fails due to small object distances. Therefore, optical flow estimates are combined with distance measurements of a Laserscanner in order to separate objects with different motions even if their distance is vanishing. Results are presented on real measurements taken in potentially harmful traffic scenarios. 1 Introduction The ARGOS project at the University of Ulm aims at a consistent dynamic description of the vehicles environment for future advanced safety applications such as automatic emergency braking, PreCrash and pedestrian safety. A Laserscanner and a video camera mounted on the test vehicle retrieve the necessary measurements of the vehicles environment [7]. The Laserscanner acquires a distance profile of the vehicles environment. Each measurement represents an object detection in 3d space. Because of the high reliability of object detection and the accurate distance measurements at a high angular resolution, the Laserscanner is well suited for object detection, tracking and classification [4]. However there are scenarios, especially in dense urban traffic where the algorithms fail. The Laserscanner tracking and classification algorithms are based on a segmentation of the measurements. The measurements are clustered with respect to their distance. Objects which are close together are therefore wrongly recognised as a single segment. Thus, object tracking and classification are bound to be incorrect. A similar problem arises in stereo vision. In [5] stereo vision is combined with optical flow estimates in order to detect moving objects even if they are close to other stationary objects. However, the approach can not differentiate between two dissimilarly moving objects. Dang et al. developed an elegant Kalman Filter implementation for object tracking using stereo vision and optical flow [3]. This algorithm uses a feature tracking approach and can be used for image segmentation based on the object dynamics. Our approach aims at a correct Laserscanner based segmentation of objects even they are close together by analysing their motion pattern in the video image

2 2 N. Kaempchen et al. domain. The segmentation criterion is therefore based on the distance between Laserscanner measurements and additionally the difference of the associated optical flow estimates in the video images. 2 Sensors A Laserscanner and a monocular camera are combined in order to enable a reliable environment recognition in distances of up to 80 m. The multi layer Laserscanner ALASCA (Automotive LAserSCAnner) of the company IBEO Automobile Sensor GmbH (Fig. 1) acquires distance profiles of the vehicles environment of up to 270 horizontal field of view at a variable scan frequency of Hz. At 10 Hz the angular resolution is 0.25 with a single shot measurement standard deviation of σ = 3 cm, thus enabling a precise distance profile of the vehicles environment. It uses four scan planes in order to compensate for the pitch angle of the ego vehicle. The Laserscanner ALASCA has been optimised for automotive application and performs robustly even in adverse weather conditions. The multi layer Laserscanner is mounted at the front bumper of the test vehicle which reduces the horizontal field of view to 180. Fig. 1. The multi layer Laserscanner ALASCA (Automotive LAserSCAnner) of the company IBEO Automobile Sensor GmbH. The monocular camera is mounted behind the windscreen beside the inner rear mirror. The camera is equipped with a 1/2 CCD chip which has a standard VGA resolution of 640x480 pixel. With a 8 mm lens a horizontal view of 44 is realised at an average angular resolution of 0.07 per pixel. In order to synchronise the sensors, the camera is triggered, when the rotating Laserscanner head is aligned with the direction of the optical axis of the camera. The sensors are calibrated in order to enable not only a temporal alignment given by the synchronisation but also a spatial alignment. By means of an accurate synchronisation and calibration, image regions can be associated directly with Laserscanner measurements. Therefore it is possible to assign certain image parts a distance, which is a major advantage of this fusion approach.

3 Spatio temporal Segmentation using Laserscanner and Video Sequences 3 3 Laserscanner based Segmentation In order to reduce the amount of data which has to be processed, the Laserscanner measurements are combined to segments. The aim of the segmentation is to generate clusters which each represent an object in reality. Optimally there is exactly one segment per object and only one object per segment. This is, however, not always possible to realise. The segments are created based on a distance criterion. Measurements with a small distance to neighbouring measurements are included in the same segment. Both the x and y components of the distance d x and d y have to be below a certain threshold θ 0. For urban scenarios a sensible choice is θ 0 = 0.7 m. Especially in urban areas where the object density is high, two objects might be so close together that all measurements on these objects are combined to a single segment. This is critical for object tracking and classification algorithms which are based on the segmentation. If the measurements of two objects are combined to one single segment the object tracking can not estimate the true velocity of the two objects which is especially severe if the objects exhibit different velocities (Fig. 2). Additionally a classification of the object type (car, truck, pedestrian, small stationary objects and large stationary object) based on the segment dimensions is bound to be incorrect. However, reducing the threshold θ 0 results in an increase of objects which are represented by several segments. This object disintegration is difficult to handle using only Laserscanner measurements. To the authors knowledge there has not yet been suggested any real time Laserscanner object tracking algorithm which is robust against a strong object disintegration in urban scenarios. Fig.2. Laserscanner based segmentation of a parking scenario at two time instances.

4 4 N. Kaempchen et al. 4 Spatio temporal Segmentation using Optical Flow In order to improve the Laserscanner based segmentation which uses a distance criterion, an additional criterion is introduced. Considering two consecutive images the optical flow can be calculated for image regions which are associated with Laserscanner measurements. Using the optical flow as an additional segmentation criterion enables the differentiation between objects of diverging lateral motions even if they are close together. The optical flow f = ( f u f v ) is calculated with a gradient based method [1, 8,6]. In automotive applications, the ego motion component of the optical flow can be high even when using short measurement intervals. Therefore, a pyramidal optical flow estimation is applied in order to account for large displacements [2]. Two spatio temporal segmentation algorithms have been developed the constructive and destructive segmentation. 4.1 Constructive Segmentation The constructive approach changes the segmentation distance threshold θ 0 depending on the similarity of the assigned optical flow. Extending the optical flow vector without loss of generality with the time dimension 1 ˆf = f 2 u + fv f u f v 1, (1) the similarity of two optical flow vectorsˆf 1 andˆf 2 is given by the angle ψ between the vectors [1] ) ψ = arccos (ˆf1 ˆf 2, with ψ [0, π]. (2) This similarity measure ψ is, however, biased towards large optical flow vectors f. Therefore the optical flow vectors are normalised, with f = 2 f, (3) f 1 + f 2 before applying equation (1) and (2). The segmentation process is performed as in the Laserscanner based approach. Two Laserscanner measurements are assigned to the same segment if their distance components d x and d y are below the threshold θ 0. However the threshold is now a function of the similarity measure ψ, with θ(ψ) = θ 0 (aψ + b), (4) where a and b are parameters of a linear transformation of ψ. The parameters a and b are chosen so that θ(ψ) is increased for similar optical flow vectors

5 Spatio temporal Segmentation using Laserscanner and Video Sequences 5 and decreased for dissimilar vectors. If there is no optical flow assigned to the Laserscanner measurement a threshold of θ(ψ) = θ 0 is chosen. This segmentation approach performs well if the optical flow vectors can be determined precisely even at the object boundaries where occlusions occur. As this could not be achieved with the chosen optical flow approach a second segmentation was developed which is more robust against inaccurate optical flow estimates. 4.2 Destructive Segmentation The destructive approach is based on the segmentation of Laserscanner measurements described in section 3. The threshold θ 0 is chosen so that the object disintegration is low. Therefore, measurements on objects which are close together, are often assigned to the same segment. In this approach the video images are used to perform a segmentation based on optical flow estimates. The Laserscanner and video based segmentation are performed individually. If the optical flow segmentation indicates the existence of several objects within the image region (c) fu (b) α (a) cameras field of view Fig. 3. Optical flow profile assigned to a Laserscanner segment. (a) shows Laserscanner measurements which are associated to the same Laserscanner segment, (b) the respective image region, (c) the horizontal optical flow component f u for the four scan layers as a function of the viewing angle α. The dotted horizontal lines indicate the α axis for the individual scan layers.

6 6 N. Kaempchen et al. fu Fig.4. Approximation of the optical flow profile by a set of linear functions. The detected object boundary is indicated with the vertical dashed lines. α of an associated Laserscanner segment, the Laserscanner segment is separated according to the optical flow segments. Fig. 3 shows a Laserscanner segment and the associated image region of a parking situation. The distant car backs out of a parking space. The optical flow estimation is attention driven and only calculated at image regions which are assigned to a Laserscanner measurement. The horizontal optical flow component f u for the four scan layers is shown in Fig. 3 (c). This optical flow profile is used for the optical flow based segmentation. The raw optical flow profile is corrupted by outliers caused by reflections or other effects which violate the assumptions of the brightness change constraint equation [6]. Therefore a median filter is applied to the optical flow estimates in order to reduce the number of outliers. The object boundaries appear in the optical flow profile as discontinuities. In order to detect these discontinuities, the profile is approximated by a set of linear functions (Fig. 4). Initially, the optical flow profile is represented by a single line segment L i. Recursively, a line segment is split into two if the maximal distance d(α, L i ) of the optical flow profile to a line segment exceeds a threshold κ, d(α, L i ) > κ( f ). (5) The threshold κ is motivated by the noise in the optical flow estimates which is a function of the magnitude of the optical flow vector f and the expected errors caused by violations of the brightness change constraint equation. The gradients m(l i, n) of the line segments L i of the individual scan layers n are combined to an averaged estimate m(l i ), after deletion of potential outliers m(l i ) = 1 m(l i, n), (6) N where N is the number of scan layers. Object boundaries are classified based on the averaged gradient of the line segments m(l i ), with n m(l i ) > m max, (7) where m max is the maximal allowable steepness for a line segment of a single rigid object.

7 Spatio temporal Segmentation using Laserscanner and Video Sequences 7 The destructive segmentation assumes objects to be rigid and that object boundaries are mainly vertical in the image domain. In the parking scenarios chosen for evaluation purposes these assumptions are perfectly met. 5 Results The presented segmentation algorithms were evaluated on parking scenarios. In all scenarios a car backs out of a parking lot. The speed of the ego vehicle varies across the scenarios, which introduces an additional optical flow component with increasing magnitude towards the image borders. Twelve scenarios were investigated with both segmentation approaches and compared to the Laserscanner segmentation. The focus was on the car which backs out and its neighbouring car. The time stamp and the position of the moving car were recorded when the two objects were first continuously separated by the Laserscanner approach. Then, the time stamps and positions for the two other approaches were noted. The average of the differences between time and position of the optical flow based approaches with respect to the Laserscanner segmentation are concluded in Table 1. Table 1. Gained time and the respective covered distance of the car backing out of the parking lot. Constructive Destructive Time [sec] Distance [m] Time [sec] Distance [m] In average the optical flow based segmentations detect the moving car as an individual object 2.2 sec (2.3 sec) earlier than the Laserscanner segmentation. This gained time corresponds to a covered distance of the car backing out of the parking lot of 1.5 m (1.6 m). The two spatio temporal segmentations perform similar in terms of an early separation of the two objects of different lateral speeds. However, the more general constructive approach exhibits a higher degree of object disintegrations. The two objects are often represented by more than two segments. This is due to inaccuracies in the optical flow estimation especially at object borders. The destructive approach is less general, as is takes only the horizontal optical flow component into account. This is, however, the main motion component of cars moving lateral to the sensors viewing direction and therefore sufficient to consider with respect to the application. The filtering and region based linear approximation of optical flow estimates enables the algorithm to be more robust against inaccuracies in the optical flow estimation. The result is a very low degree of object disintegration.

8 8 N. Kaempchen et al. Further examination of the results exhibited that the performance depends on two main factors independently of the chosen algorithm. First, the performance decreases with increasing velocity of the ego vehicle as the optical flow artefacts and the noise raises and therefore the SNR decreases. Second, the performance depends on the velocity difference tangential to the viewing direction between the close objects. The higher the velocity difference the better the performance. In the scenario of a car backing out of a parking lot, the performance depends directly on its velocity which varies in the experiments between 1 and 9 km/h. 6 Conclusion Two spatio temporal segmentation approaches have been presented. Based on Laserscanner measurements and optical flow estimates of associated image regions a robust segmentation of objects is enabled even if objects are close together. In potentially harmful situations the correct segmentation allows a precise tracking of moving objects. The accurate segmentation and therefore tracking is an essential prerequisite for a reliable prediction of objects in dynamic scenarios for active safety systems in future cars such as automatic emergency braking. References 1. J. Barron, D. Fleet, S. Beauchemin, and T. Burkitt, Performance of optical flow techniques, in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 1992, pp J.-Y. Bouguet, Pyramidal implementation of the lucas kanade feature tracker, Intel Corporation, Microprocessor Research Labs, Tech. Rep., T. Dang, C. Hoffmann, and C. Stiller, Fusing optical flow and stereo disparity for object tracking, in IEEE 5th International Conference on Intelligent Transportation Systems, Singapore, September 2002, pp K. C. Fuerstenberg and K. C. Dietmayer, Object tracking and classification for multiple active safety and comfort applications using multilayer laserscanners, in Proceedings of IV 2004, IEEE Intelligent Vehicles Symposium, Parma, Italy, June 2004, accepted. 5. S. Heinrich, Real time fusion of motion and stereo using flow/depth constraint for fast obstacle detection, in Pattern Recognition, 24th DAGM Symposium, ser. Lecture Notes in Computer Science, L. J. V. Gool, Ed., no Zurich, Switzerland: Springer 2002, ISBN X, September 2002, pp B. Jähne, H. Haußecker, and P. Geißler, Eds., Handbook of Computer Vision and Applications. Academic Press, ISBN , May N. Kaempchen, K. Fuerstenberg, A. Skibicki, and K. Dietmayer, Sensor fusion for multiple automotive active safety and comfort applications, in 8th International Forum on Advanced Microsystems for Automotive Applications, Berlin, Germany, March 2004, pp J. Shi and C. Tomasi, Good features to track, in IEEE Conference on Computer Vision and Pattern Recognition, 1994, pp