Evaluation of a laser-based reference system for ADAS

Transcription

1 23 rd ITS World Congress, Melbourne, Australia, October 2016 Paper number ITS- EU-TP0045 Evaluation of a laser-based reference system for ADAS N. Steinhardt 1*, S. Kaufmann 2, S. Rebhan 1, U. Lages 2, Y. Noutangnin 1, C. Goerick 1, C. Ahlers 2 1. Honda Research Institute Europe (HRI-EU) GmbH, Carl-Legien-Strasse 30, Offenbach, Germany Nico.Steinhardt@honda-ri.de Ibeo Automotive Systems GmbH Merkurring 60-62, Hamburg Germany stefan.kaufmann@ibeo-as.com Abstract The evaluation and verification of advanced driver assistance systems (ADAS) is a crucial element for guaranteeing safe operation of such systems on the road. To cover a wide variety of real-world situations, ground-truth data needs to be generated for large datasets, rendering complete manual annotation infeasible. However, when employing automatic ground-truth reference systems, the performance and accuracy of these systems becomes subject to validation itself. In this paper, we present a three-step methodology for characterizing a 360 degree reference system based on vehicle-mounted laser scanners, which is widely used for automatized labelling of data. Furthermore, a high-accuracy ground-truth reference system (GTRS) is presented that is used to determine the object classification and tracking performance as well as the position accuracy of the device-under-test (DuT). This methodology comprises the accuracy evaluation by conducting measurements series with increasing complexity starting with idealized objects up to complex real-world traffic situations. By doing so, the cause of accuracy changes can be pinned down to properties of the object or scene constellation allowing a deeper understanding of the performance that can be expected from the DuT using real world data. In this paper, the first step of the methodology is presented, that shows the abstraction from idealized to real traffic participants. Keywords: Ground-truth measurement, object validation, LIDAR

2 Introduction Perception of the environment is crucial for advanced driver assistance systems (ADAS). Modern vehicles are equipped with various sensors which extract information and provide those to the ADAS. The development of such sensors and related perception algorithms, as well as the development of ADAS functions, requires reference data for testing and validation. Normally, a lot of manual labelling work is necessary for creating this reference data and therefore typically only a few reference data sets are generated. However, having a large database of reference scenarios speeds up the development process and improves the perception quality of environmental sensors and algorithms. This paper presents an approach for generating reference data with a minimal amount of manual labelling work. This data will be used to evaluate the quality of the Ibeo reference system [1]. Related work can be found in [2] and [3] with a slightly different focus on methodology. The approach is based on the Ibeo reference system consisting of a laser scanner fusion system with 360 field of view which is mounted on a passenger car (Figure 1). This system generates data for dynamic objects in the vehicle environment, with relevant properties such as object position, yaw angle, velocity, yaw rate and object class. Figure 1: Ibeo Reference System Setup Using the aforementioned ground-truth reference (GTRS) data, we will describe a systematic approach for evaluating the dynamic object tracking performance of the Ibeo reference system. The reference system under evaluation is referred as DuT in the following. Methodology The core idea of the evaluation methodology is to stepwise derive the real-world accuracy of the DuT and to identify all influences contained. This is done by applying a top-down procedure, starting with static idealized objects with exactly known properties, and advancing towards more and more realistic setups. The setup of all scenarios are measured by a 2

3 ground-truth reference system (GTRS), which allows identifying both the basic DuT accuracy as well as the additional uncertainties of each step towards more realistic scenarios. For obtaining more statistical significant measurements, the measurements are taken using two different DuTs, which both are series vehicles equipped with the Ibeo laser-based reference system. The approach is organized in 3 phases: Phase 1: Identification of base DuT accuracy, taking the step from idealized static objects to real static objects, as well as from real static objects to real dynamic objects. Creation of statistics for object classification, object position/velocity accuracy. Measurements on test tracks with exactly known geometric properties. Phase 2: Application of the statistics to predefined traffic situations on real roads, taking the step from idealized traffic situations to real traffic situations on highways and rural roads. Creation of statistics for scenario detection (e.g., cut in, cut out and overtaking). Phase 3: Application of the statistics from Phases 1 and 2 for taking the step to complex urban traffic situations. Creation of statistics for complex scenario understanding. In the scope of this paper, only work of Phase 1 will be presented. The evaluation of the DuT is done starting from idealized objects to real objects. The following static and dynamic scenarios are planned for Phase 1 and described in detail in the following sections: 1. Setup of static scenarios using object boxes (rectangular cubes) with the same maximum dimensions as real traffic participants (i.e., pedestrian, motorbike and car). These tests will provide statistics for the maximum achievable static position accuracy of the DuT on targets with precisely known geometric properties. These tests will be conducted from different distances, covering the entire detection range (i.e., from short range up to the maximum detection range for the DuT). 2. Repetition of all step 1 measurements using the corresponding real traffic participants. This test will provide statistics for the influences of a realistic vehicle shape on the static position accuracy of the DuT compared to idealized targets, and therefore perform the first step from ideal to real objects. 3. Setup of dynamic traffic situations using the same targets as in step 1 and 2 on a test track with predefined and geometrically known road setup (e.g., oncoming traffic and intersections). The measurements will include both static, constantly moving and accelerating / braking targets, by this taking the second step from static to dynamic objects. These tests will provide statistics about velocity accuracy and dynamic position accuracy of the DuT. 4. All of the tests are used to obtain statistics for classification reliability. 3

4 Approach for ground-truth data generation For the evaluation of the dynamic object tracking provided by the DuT, a ground truth reference system (GTRS) is used, as illustrated in Figure 2. It consists of a static, external measurement setup and provides an accuracy of 20mm for the evaluation of the DuT data. The accuracy calculation includes the GTRS accuracy of 7mm (noise average) and accuracy of used measurement equipment to measure GTRS relative positions and box positions to ground truth (Disto Leica D5 with 3mm accuracy and a meter stick with an estimated accuracy of 10mm). The GTRS consists of two long-range distance sensors (SICK DML40-2) and two laptops for recording the data. In addition, video streams are recorded for scene understanding. As the test setups require different positioning of the long-range scanners (parallel setup, orthogonal setup), the relative position of the scanners is determined using geometric methods (e.g. triangulation) prior to the measurements. The accuracy estimation of the different GTRS setups will be shown later on in the corresponding chapters. In addition, Garmin GPS receivers are used for time synchronization of the GTRS as well as in the DuTs. This ensures that the local timestamps will be synchronized to UTC, so that the measurements will all be taken in the same timeframe and can be easily synchronized in the data evaluation phase. The GPS position will not be evaluated, as its accuracy is even below the expected DuT accuracy. For better comparability of the data, all measurements from the GTRS as well as from the DUT are projected to the same reference point: the center of the rear axle of any vehicle. Known measurement offsets (e.g. mounting position of the triple reflector or size of the bumper), viewing direction onto the vehicle (front, back) and their dynamic changes are also compensated. Garmin GPS Data recording Time synchronization Cameras Data recording Laptops Figure 2: Ground Truth Reference System (GTRS) Step 1: Static position accuracy for ideal objects Following the aforementioned methodology, first tests are performed to obtain the object position accuracy of the DuT for ideal objects. The DuT can classify six kinds of object types: pedestrian, motorbike, car, truck, unknown small and unknown big. Since the object classification is strongly coupled to the size of the object, object boxes have been used with 4

5 different sizes for a motorbike and a car. As shown in Figure 3, a Honda CBF1000 motorcycle (2.165m x 0.732m, incl. rider) and a Honda Civic (4.285 x 1.770m) have been used as real objects, and the idealized boxes have been designed accordingly. For the pedestrian a cylinder shaped tube (h=2.0m, d=0.6m) has been used. The truck instead has been chosen to already have a shape close to a box. By this, the creation of a very large and unhandy box has been avoided. Figure 3: Setup of static position accuracy test with ideal object boxes The static position accuracy tests have been performed with different distances and orientations of the ideal boxes. A list of all the tests performed on the ideal boxes is given in Table 1. 5

6 Table 1: Test scenarios for static position accuracy tests Object orientation DuT Position 7m 20m 50m 100m Pedestrian Motorbike Motorbike Car Car Truck Truck Measurement Principle The measurement principle for obtaining statistics about the static position accuracy of the DuT is indicated in Figure 5. Two GTRS scanners have been used to get distance information of the DuT relative to the scanners. The geometrical uncertainty of the GTRS in this measurement setup is 20mm as described above. Two triple glass reflectors have been diagonally attached to the DuT for obtaining two well-defined points to be measured by the GTRS (see Figure 4), so both the position as well as the heading angle of the DuT relative to the objects can be determined by measurement triangulation. All other distances shown in the figure (e.g., the distance between the GTRS scanners and the distances to the object) have been measured by using a hand held laser distance measurement device. As listed in Table 1, the setup was repeated for different object classes, different object orientations and different distances (d_dut) to be able to obtain statistics. Figure 4: Glass triple reflector mounted on the roof of the DuT (left) and GTRS scanner mounted on a tripod (right) 6

7 d d_lf d_rf d_lr d_rr d_dut DUT Triple glass reflectors DUT Setup 1 Setup 2 Figure 5: Measurement principle for static position accuracy tests Step 2: Static position accuracy for real objects After considering object boxes, real objects have been used and all the setups of step 1 have been repeated. Step 3: Dynamic position/velocity accuracy for realistic traffic situations The first two steps were dedicated to obtain statistics for static traffic situations. As described in the methodology, the step to dynamic measurements is taken in this chapter. The values of interest are again position accuracy as well as the accuracy of the object velocity estimated by the DuT. The setup of the GTRS for longitudinal measurements is shown in Figure 6. The GTRS measured the longitudinal object position with a rate of 200 Hz. The velocity is computed using the derivative of the GTRS measurements. For the final statistical evaluations it will be additionally checked if this value requires filtering. The uncertainty of the GTRS setup is at 10mm (no meter stick used in this setup). Those measurements are used as ground truth data and will be compared to the velocity data provided by the DuT. DUT Object GTRS Figure 6: Longitudinal velocity measurements using the GTRS 7

8 Setups for longitudinal motion The setups for longitudinal motion tracking are shown in Table 2, the measurement setup is shown in Figure 6. The geometrical uncertainty of the GTRS in this measurement setup is 10mm. A subset of combinations of DuT speed and object speed were considered in this experiment. The methodology of selection targets two criteria: On the one hand, situations that are generally known to be difficult to handle for object tracking algorithms (e.g., very large and very low velocity differences, strong changes of velocity) shall be contained, on the other hand, these manoeuvres shall be measured in a context in which they are most likely to occur in real-world traffic. Therefore, the following scenarios were selected: Setups with typical inner city speeds (30 km/h, 50 km/h). In addition 100 km/h and dynamic tests (acceleration up to 100 km/h and hard braking to full stop) Setups with both high and low relative velocity between DuT and Object for overtaking scenarios Low and high relative velocities for oncoming traffic situations Negative speed values in the table indicate oncoming traffic relative to the DuT. The columns and rows with speed ranges (e.g km/h) indicate dynamic driving behaviour. Dynamic tests have been performed for different driving directions of DuT and object and for the same driving directions. In case of different driving directions, the DuT and object where passing each other with high speed on neighbouring lanes and a full braking manoeuver was done by the drivers to stop the vehicles in front of each other. After stopping, the vehicles again accelerated to high speed. Those tests were performed to obtain values for position and speed accuracy of the detected objects for high relative accelerations. For setups with the same driving direction and speed ranges, the vehicles performed an overtaking manoeuver multiple times (one vehicle accelerated, while the other was braking and vice versa). 8

9 Table 2: Overview about performed longitudinal tracking tests DuT Speed 0 km/h Object Speed km/h km/h km/h km/h Pedestrian 0 km/h km/h Motorbike km/h km/h km/h km/h km/h km/h km/h km/h Car km/h km/h km/h km/h km/h km/h km/h km/h Truck km/h km/h km/h km/h km/h km/h km/h Setups for lateral motion As mentioned before, the tracking performance for lateral motion is evaluated with a different setup since the GTRS can only track one-dimensional motion. The setup itself is illustrated in Figure 7. The geometrical uncertainty of the GTRS in this measurement setup is 10mm. The performed tests are listed in Table 3. 9

10 DUT Figure 7: Setup of the intersection scenarios Again the selection of the test setups followed the methodology of having low and high relative velocities between the DuT and the objects. For the static object tests (speed 0 km/h) the objects have been placed close to the intersection. For the other tests, the objects stopped in front of the intersection to let the DuT vehicle pass. Table 3: Overview about performed lateral tracking tests Object Speed DuT Speed 30 km/h 50 km/h 100 km/h km/h Pedestrian 5 km/h Motorbike 0 km/h km/h km/h Car 0 km/h km/h km/h Truck 0 km/h km/h An example of the lateral tracking tests approaching a truck which is stopping at the intersection is shown in Figure 8. The figure shows the recorded data from the DuT for three different time frames. 10

11 Truck Truck DuT DuT Truck DuT Figure 8: Lateral tracking test. DuT speed: 100km/h, truck 30 km/h stopping at the intersection Preliminary Experimental Results Even though the evaluation of the experimental results is still ongoing, we present some preliminary results in this section below. Figure 9: Comparison of object position measured from DuT and from GTRS The results for the setup measuring a real car (test scenario ) for large acceleration values is shown in Figure 9. As described before, both the DuT and the car were overtaking each other multiple times. As already mentioned, all measurements refer to the center of the rear axle of the vehicles. In the bottom figure, the error is plotted. The Figure shows some peaks in the position error for certain time ranges. Those peaks have different reasons: Peak 2 and 3 are corresponding to time ranges (typically a few milliseconds), where no GTRS signal was available (since the scanner did not hit the 11

12 reflector) or other disturbances were introduced the reference measurement. These parts of the measurements will not be considered for the overall performance evaluation. To identify those parts, characteristic features are used. In this case the used characteristic is the derivative of the target positions measured by the GTRS. If the reference GTRS position is changing faster than the physical capabilities of the target, the data point is marked as invalid. On the other hand, Peak 1, 4, 5 and 6 as shown in Figure 9 are corresponding to real differences between GTRS and DuT. In these cases the GTRS data are well-formed and the DuT measured a different distance to the object than the GTRS. Those are the peaks that have to be considered in the overall performance statistics later on and will be analysed to determine the accuracy of the DuT. The resulting statistic values in test scenario without the false GTRS Data Peaks 2 and 3 was m in maximum (Peak 4 in Figure 9) and m in minimum (Peak 1 in Figure 9), the mean deviation between DuT and GTRS was m and the standard deviation was m. Conclusion and Outlook A top-down methodology for evaluating a laser-based reference system has been shown. Relevant scenarios have been identified and utilized for generating data on a proving ground using two different DuTs. In the single experiment that was shown, the accuracy of the DuT would have been sufficiently high for its use as a reference system after the removal of the outliers. As soon as the complete available data is processed, more profound statistics about object tracking position and velocity accuracy will be generated from the static and dynamic scenarios shown in this paper. Additional parameters like dependencies from velocity, acceleration and distance will be identified and taken into account. These results will then be the basis for the real-world driving scenarios in project phase 2. As the measurements were conducted with well-known targets, the verified, true positive object detections alone were considered for the classification reliability. Statistics about object detection reliability will be part of project phase 3. References 1. M. Spencer, R. Katz, U. Lages (2014), Forward-Backward Object Tracking for Generation of Reference Scenarios Based on Laser Scan Data, In Proceedings 21st World Congress on ITS, Detroit 2. Wulf, O.; Nuchter, A.; Hertzberg, J.; Wagner, B., "Ground truth evaluation of large urban 6D SLAM," in Intelligent Robots and Systems, IROS IEEE/RSJ International Conference on, vol., no., pp , Oct Nov Blanco, J.L; Moreno, F.-A.; Gonzalez, J. (2009) A collection of outdoor robotic datasets with centimeter-accuracy ground truth, In Autonomous Robots 27(4)