SAFESPOT INTEGRATED PROJECT - IST IP DELIVERABLE SAFEPROBE. In-vehicle platform test results

Transcription

1 SAFESPOT INTEGRATED PROJECT - IST IP DELIVERABLE SP1 In-vehicle platform test results Deliverable No. (use the number indicated on technical annex) D1.5.2 SubProject No. SP1 SubProject Title Workpackage No. WP5 Workpackage Title Test and validation Task No. T1.5.3 Task Title In-vehicle platform validation Authors (per company, if more than one company provide it together) Status (F: final; D: draft; RD: revised draft): Cosenza Stefano, Andrea Saroldi, CRF; Andreas Ekfjorden, VOLVO; Florian Ahlers, IBEO; Christian Zott, BOSCH; Fa Zhang, Hongjun Pu, CA; Panagiotis Lytrivis, ICCS F Version No: 1.2 File Name: SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc Planned Date of submission according to TA: 31/03/09 Issue Date: 23/04/09 Project start date and duration 01 February 2006, 48 Months SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 1 of 72

2 Revision Log Version Date Reason Name and Company Initial skeleton Christian Zott Inputs about DAQ from Continental Added Volvo tests (Gateway test still missing) Hongjun Pu Andreas Ekfjorden Added Volvo gateway tests Björn Svedlund Added ICCS Contribution about SR Added contribution from Bosch, IBEO and CRF Added contribution from CRF Added contribution about radar sensor from CRF Added contribution about OR tests from CRF Final version Harmonisation of contributions, version ready for Peer Review Integration of Peer Review comments Panagiotis Lytrivis Christian Zott, Florian Ahlers, Stefano Cosenza Stefano Cosenza Andrea Saroldi Andrea Saroldi Stefano Cosenza Andrea Saroldi Andrea Saroldi, Stefano Cosenza SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 2 of 72

3 Abbreviation List ADAS CAN CPDF DAQ FW GPS HW IP LDM LRR NTP OR SP SR SW UDP VANET Advanced Driver Assistance Systems Controller Area Network Cooperative Data-Fusion Data Acquisition Framework Global Positioning System Hardware Information Provider Local Dynamic Map Long Range Radar Network Time Protocol Object Refinement Sub-Project Situation Refinement Software User Datagram Protocol Vehicular Ad-Hoc Network SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 3 of 72

4 Table of contents Revision Log...2 Abbreviation List...3 Table of contents...4 List of Figures...5 List of Tables...5 EXECUTIVE SUMMARY Introduction Innovation and Contribution to the SAFESPOT Objectives Methodology Deliverable structure Overview of test cases In-lab test cases Functional Component testing Integration testing In-vehicle test cases Functional Component testing...errore. Il segnalibro non è definito SR testing In-lab test results Functional Component Testing DAQ OR SR IP and LDM Functional Integration Testing VANET integration Positioning integration Synchronization NTP Shutdown sequence In-vehicle validation Functional component testing Gateways POSITIONING VANET Laserscanner, CPDF LRR OR Conclusions References Annexes List of test cases (completed test forms including results) Test cases from ICCS Test cases from CRF Test cases from Continental Test cases from IBEO Test cases from BOSCH Requirement fulfilment tables...62 SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 4 of 72

5 List of Figures Figure 1: The V-model...8 Figure 2: Phases of the validation plan...8 Figure 3: Processing Time Measurement...10 Figure 4: CRF test vehicles...12 Figure 5: time distribution within the OR module...15 Figure 6: Filled table in the LDM...16 Figure 7: SR functionality testing...17 Figure 8: Input data to SR in the vehicle (SR_In.txt)...17 Figure 9: Output data of SR in the vehicle (SR_Out.txt)...18 Figure 10: Validation of SR with the use of SafeFusion simulator...18 Figure 11: Timing test on Gateway...24 Figure 12: Gateway repetition time rate...25 Figure 13: Gateway repetition time rate...25 Figure 14: drift of the vehicle while driving along a curve...27 Figure 15: Object distribution in front of the radar during the functional test...28 List of Tables Table 1: Framework and IP test cases...9 Table 2: LRR performances...29 SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 5 of 72

6 EXECUTIVE SUMMARY This document describes the results of the functional tests of the subproject. The main objective of this work is to report about the results from testing and validating the functionalities of SAFESPOT main component and modules as SW and HW. Another objective is to show to which extent the specifications defined in previous deliverables could be met. The testing has been structured by test cases, which are based on the existing use cases and in accordance with the SCOVA sub-project. The validation procedure can be divided into three phases: the in-lab testing, the invehicle platform validation and the performance analysis phase. As a consequence the chapters of this test report are organized according to these phases. The validating phase is the core part of the deliverable SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 6 of 72

7 1. Introduction The sub-project is responsible for the vehicular platforms to be used in SAFESPOT. The aim of this document is to present the test report of SP1 tests concentrating on functional testing. This document is accompanied by D1.5.3 Performance analysis which focuses more on quantitative test results corresponding to non-functional requirements. This deliverable is strongly connected with the D1.5.1 [5] that describes the SP1 test and validation plan and the test cases referred to in this report. The requirements and specifications of this sub-project have been provided in deliverables D1.2.3 [1], D1.3.1 [2] and D1.3.2 [3]. Finally, it should be mentioned that this deliverable is also strongly connected with the D1.4.2 [4] that describes the SP1 test sites, the demonstrator vehicles, the test bed components etc Innovation and Contribution to the SAFESPOT Objectives The compatibility and function integration is of paramount importance when a complex architecture is brought to work and provide output as for the SAFESPOT project. It is, hence, necessary to define procedures and criteria to evaluate if the different modules are able not only to interoperate, but also to provide usable data by the platform. The functional validation and integration of the SAFESPOT component is the first preparatory phase to the second testing level that is the validation of the platform with the data analysis in term of respect of the constrain and satisfaction of the requirements. In this documents the alignment among the versions, correct start up and shut down of the system, and the properly calibration of the different modules has been investigated and reported Methodology The proposed method is based on the methodology implemented in the PReVAL subproject [5]. PReVAL provided PReVENT with a harmonised evaluation framework, defined a methodology to be used in the impact assessment of various applications and applied the methodology to a set of given use cases. PReVAL also proposed a set of recommendations for future research needed for the impact assessment, best practices and a basis of guidelines for the use of intelligent vehicle development. The PReVAL method has the following advantages: This method achieved broad consensus within PREVENT, therefore we can consider it as a reference; PREVENT itself is based on the methods proposed in CONVERGE and APRSOSYS which constitute the foundations of evaluation; It is the method most recently applied to ADAS inheriting of the most successful reflections in this field. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 7 of 72

8 The methodology for the validation plan that is described in this deliverable is based as the PReVAL s approach on the V-model. The V-model is a well known model that is followed in validation procedure. In the following figure this model is depicted. Figure 1: The V-model The validation report can be separated in three phases: in-lab testing, in-vehicle testing, and performance analysis. These phases can be consecutive Deliverable structure Figure 2: Phases of the validation plan Chapter 2 focuses on the testing results performed inside the laboratory. The results from testing the FW and the other threads (DAQ, OR, SR, IP) are described there. In chapter 3 and 4 the findings from testing platforms in the lab and in vehicles can be found. The test reports for each test case as specified in D1.5.1 [5] are listed in annex A, in the formal SAFESPOT format (test forms). A table showing the grade of fulfilment of s requirements can be found in annex B. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 8 of 72

9 2. Overview of test cases 2.1. In-lab test cases Functional Component testing Framework, IP and LDM For component testing of the framework, the information provider (IP) and the integration with the LDM there are 2 in-lab test cases (see D1.5.1 appendix A, and table below). Test Case number Test Case Title Parameters collected Bosch_1 Bosch_2 Framework and LDM update correctness Framework and LDM update timing performance Table 1: Framework and IP test cases Logged Ethernet (wireshark), framework (text files) and LDM updates (pgadmin, OpenJump, logging files) Task s execution times [ms] using Window s high-resolution QueryPerformanceCounter Main Setup An IBM ThinkPad T41 notebook (Pentium M, 1600MHz, 512MB RAM, which is comparable with ebox, with Pentium M, and 1GB RAM) acted as mainpc. It hosted the SP1 Framework including the SP3 PG-LDM. The second notebook was used as a UDP-player and was connected via Ethernet to the mainpc. The tested Framework version was 8.4 in combination with the PG-LDM-API v This PG-LDM-API is compatible to the LDM-schema version The version 2v10a of the Positioning-UDP-Player was used to stimulate the mainpc. The Positioning-UDP-Player sent the messages with 12.5Hz. The Framework was set to trigger the data fusion chain every 160msec. Since both tests have the main focus on the Framework with Information Provider (IP) and their interaction with the LDM dlls, constant outputs were generated for Object Refinement (OR) and Situation Refinement (SR). The OR-dll outputs a list of 5 objects including two vehicles and three unknown objects. The SR-dll outputs the (constant) lane assignment results for the OR-outputs. These two dlls did not contain any algorithms and therefore consumed only constant processing time at a low level (few milliseconds). In this test configuration the following LDM-tables were modified by the framework: alongroadelement SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 9 of 72

10 egomotorvehicle motorvehicle uo The procedure of the processing time measurement is displayed in the figure below. Beside DAQ proc time, OR proc time and SR proc time, all other measured processing times were measured starting from the arrival of a new Positioning-UDPmessage. DAQ proc time processing time of the DAQ module for a Positioning-UDP-message OR proc time processing time of the Information Provider (IP_or) module for updating the OR-results in the LDM, including the processing time on the LDM side. SR proc time processing time of the Information Provider (IP_sr) module for updating the SR-results in the LDM, including the processing time on the LDM side. Figure 3: Processing Time Measurement The results and conclusions of the time measurements can be found in D SR testing The SR.dll version 8.2, which was the latest at the moment of writing the present document, was used for functional testing in the laboratory. Testing was done with version 8.2 of the Framework (with latest compatible versions of DAQ.dll and OR.dll), LDM schema version and LDM API In the lab, TUC positioning player, CRF gateway player, and IBEO players were used for functional tests (latest versions compatible with FW version 8.2). SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 10 of 72

11 DAQ The DAQ (Data Acquisition) module is devoted to the pre-processing of the UDP datagram sent from other components such as Vehicle Gateway, VANET, Positioning PC, Laser Scanner, Camera, etc., into the main PC. The functionality of DAQ module can be tested in laboratory using Data Players feeding the UDP data streams in the framework. As soon as the simulated messages are captured by the framework, they will be forwarded to DAQ module in which the raw messages are decoded and interpreted. During the test a logging file (DAQ_Out.txt) will be generated. A comparison of the original messages provided by the Data Player with the logging file can clearly indicate whether the DAQ module works correctly in the framework. In this sense, 4 test cases have been defined for the functional laboratory test of the DAQ module and described in the Annex of this document as well as in D1.5.1 [5]. OR The OR (Object Refinement) has the tasks to take part of the input provided in the shared memory by the DAQ, process them and provide the output to the following module: the SR. The OR functionalities, as for the DAQ, can be tested in a laboratory using data players that feed the framework with pre recorded data taken either from real or simulated scenarios. The data processing can be performed using different approaches: one is to consider the txt file generated by the framework application, while it is running in debug mode selecting the option LOG OR, and using, in a second moment, a CRF proprietary and dedicated tool, named OR tester to visualize the data and the result of the OR algorithm (the OR TESTER has been already described within the D1.4.2). A second opportunity is to verify the alignment between the data sent out from the DAQ, the configuration parameter of the OR and the input/output of the OR itself. CRF is the responsible for maintaining and updating this module in relation with the update of the Framework. The performed laboratory test cases are described and reported in the annex of this document. DAQ Integration testing No test cases have been defined for stand-alone test of DAQ, since the DAQ module can only run as a thread of the frame work. In this sense, the functional tests are in fact integration tests (in the framework) as well. OR As already indicated in the DAQ paragraph, the OR represents a module that is fully integrated within the SP1 framework. In particular the OR module needs the Framework and the DAQ to be able to provide any type of output to the next SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 11 of 72

12 components. In every case, the perfect compatibility with the framework is required and in this sense no issues can be raised on the on going Framework version (8.3) In-vehicle test cases SR testing Except for testing the SR.dll in the lab, dedicated tests were carried out by CRF and ICCS colleagues on 9 th and 10 th of March on the Orbassano test track. Also here the SR.dll version 8.2 together with version 8.2 of the Framework (with latest compatible versions of DAQ.dll and OR.dll), LDM schema version and LDM API were used. For the above tests, two SAFESPOT CRF vehicles were used; the demonstrator and the SCOVA one (see Figure 4). Figure 4: CRF test vehicles The Testing of the SR cycle was carried out in the lab with data gathered during the test cases in Orbassano. That is, real data (with timestamps) stored in SR_In and SR_Out txt files, exploiting the logging properties of the FW, were used for testing the performance. During the tests all SAFESPOT HW and SW systems were running on the CRF demo vehicles (e.g. main PC FW & LDM, positioning PC, application PC, VANET router). The average duration of one SR cycle was measured ms. More details can be found in [6] OR Testing CRF role within SAFESPOT project provided the opportunity to carry on a parallel activity of development and testing on the vehicles. The installation of the dll files on SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 12 of 72

13 the vehicles represents the last step of an activity of coding, in laboratory debugging, with the help of the data players, and in lab verification on the data processed by the OR. The final step is represented by the installation of the debugged files on the vehicles, to verify that the compatibility and performance of the component is maintained also within real test conditions. The in-vehicle test had the objective to capture data to perform a more extensive and deeper off line elaboration of the results. The first parameters, taken into account and verified in the post processing phase, have been: the integrity of the data with respect to the scenario tested and moreover the performance in time of the OR module. The vehicles adopted for the in vehicle testing are represented by the official SAFESPOT vehicles provided by CRF: and SCOVA. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 13 of 72

14 3. In-lab test results 3.1. Functional Component Testing DAQ DAQ is executed within the process provided by the framework. So it can be tested only under a running framework. Further, the DAQ can be tested in real operation of a SAFESPOT-vehicle, for example in one of the test sites. While the functionality of DAQ can be sufficiently tested in laboratory using Data Player, field-tests may validate the over all system performance, stability and reliability. In this section, the results of laboratory tests are reported. Functional tests of DAQ have been conducted for the DAQ as integrated part of the framework v8.2. The following Data Players were used: SAFESPOT IBEO SEND Tool as Laser Scanner Data Player in Test Case 1 CRF Gateway 5 as Data Player for VehicleStatus, RadarSensorObjects, PowerOffCountdown in Test Case 2 UC UDP-Position-Player_2v10a as Data Player for ego positioning in Test Case 3 Vanet2UDP - UDP2Vanet Player 2.14 as Data Player for all VANET messages in Test Case 4 For each test case, desired results as described in D1.5.1 [5] have been achieved. By comparing the number, time stamps and contents (spot test), the correctness and performance of DAQ have been verified OR The OR module is integrated with the SP1 Framework program. It is able to process data captured by the framework, pre-processed by the DAQ, and provided as obstacle information to shared memory after processing by OR. Each OR release and any further development has been verified and tested using different data player provided by each partner. The possibility to use SAFESPOT vehicles has been used to gather data from real test site scenario and replay them within a specific tester program and the OR to evaluate if the real conditions are verified and to replicate possible inconsistencies. Moreover, with lab tests on simulated or real logged data it is possible to analyse the time used by the different functions in order to understand where optimisation is needed. a. Test Case CRF-8 - Simulated Front Obstacle Running the OR function in the lab, called by a specific tester program, and fed with a mixture of real and simulated data, is a way to evaluate the time performances of the OR function and understand where the processing time is used. By running the Test SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 14 of 72

15 Case CRF-08 with simulated obstacle data and real position data, the processing time has been analysed for every data processing function inside OR. The following diagram shows the splitting of the mean processing time between the processing functions inside OR. Processing time per function Object Validity Vanet2ORConverted Laser2ORConverted Radar2ORConverted Data Fusion Object MaintenanceObject Validity Temporal Alignment Spatial Alignment Vanet2ORFused Laser2ORFused Radar2ORFused EgoVehicle_TemporalAlignment Vanet_TemporalAlignment Laser_TemporalAlignment Radar_TemporalAlignment DAF Fused2Converted ObjectsMissing Figure 5: time distribution within the OR module As can be seen from the picture, inside a total of 10 ms mean processing time, the most time demanding set of functions is the one related to temporal alignment, where each object is moved in space considering the time elapsed since it has been measured. After this set, the group of functions related to spatial alignment and to data-fusion, are the most demanding ones. In fact, the real data-fusion process is only a part of the overall processing to be done by OR. However, since 10 ms is the mean total processing time for OR, for the moment no further optimisation is needed. b. Test Case CRF-11 - Lab testing with data-players Furthermore, tests with data-players are performed in the lab before going to the vehicle with each new delivery of Framework program or OR function. In fact, it is possible in the lab to check the integration between the different SW functions called by the OR. The test is considered successful when, at the end of the processing, data are stored from the Framework program in the LDM. This test is executed by feeding the Framework in the lab with players that send the input information arriving from Positioning, VANET, Gateway (including radar), and Laser. This information is acquired by the DAQ module and then passed to the OR function, which has to generate fused-obstacle information that then has to be put by IP module into the motorvehicle table inside LDM. However, since inside OR there is a set of checks that do not consider objects that are too far in time or space, these constraints have to be relaxed in order for the OR to run in the lab with data-players. In fact, this simulation is usually executed in the SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 15 of 72

16 lab with two PCs, one running the data-players and the other running the Framework with OR function, and the two PCs are not necessarily synchronised as all the PC in the vehicle should be. Moreover, the position of the ego-vehicle generated by the Position data-player has to be nears to the position of the other vehicles generated by the VANET player, otherwise the other vehicles are not considered because they are too far in space. The following picture shows the presence of VANET objects inside motorvehicle table. Figure 6: Filled table in the LDM c. Test Case CRF-6 - Ego-vehicle behind second vehicle (vehicle/lab) Before going on the road and after tests with data-players, the functionality of the OR are tested in the laboratory with real data. This is an important step, because in the lab it is possible to better analyse what is happening having the possibility to modify the SW and test it again on the same data. In order to do so, CRF has developed specific tools to feed the OR function with real data logged on the road. This SW program, called tester, is used to verify the main functionalities, evaluate possible alternatives, and define parameters used by the algorithms. The OR program has therefore been tested in the lab on real data where the egovehicle is following a second vehicle on the test track. After data processing, the output of OR correctly shows one front obstacle moving in the same direction. The distance from the front moving obstacle is coherent with relative speed. This test will later be repeated on the vehicle with data-processing running on-board (see deliverable D1.5.3 on "Performance Analysis", test case CRF-06). SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 16 of 72

17 SR The SR_In.txt and SR_Out.txt files logged with the players in the lab showed the correct functionality of SR. They also proved that the data fusion chain was working properly in lab environment with stand alone players. The txt files were not checked for correctness at this time but only that they provided the expected output. The next step was to test the SR functionality of the data chain inside the vehicle. To check this functionality both the SafeFusion simulator and the txt files from the FW were used. Figure 7: SR functionality testing The chain was working properly also in the vehicles and provided the SR_In.txt and SR_Out.txt files from the logging capabilities of the FW. These txt files can be seen in the figures below. Figure 8: Input data to SR in the vehicle (SR_In.txt) SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 17 of 72

18 Figure 9: Output data of SR in the vehicle (SR_Out.txt) Additionally with the usage of the SafeFusion simulator the correct functionality was validated. For example, SafeFusion inspected the LDM and visualized the trajectories stored in the trajectory table, which were previously calculated inside SR with input data from DAQ, OR and static map data from the LDM (see Figure 10). Figure 10: Validation of SR with the use of SafeFusion simulator IP and LDM For the cited SW versions no deviations could be found when comparing the logged data at the different stages from Ethernet (Wireshark log) via framework logs (txt files) SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 18 of 72

19 to LDM entries. However, it should be noted that no automatic tools have been applied for data comparison. Furthermore, at the time of report writing, neither 100% of the input UDP messages nor 100% of the output LDM tables and its attributes could be checked, i.e. the correctness testing should be understood as sampling inspection. The multithreading of the core data fusion threads DAQ, OR, SR proved to work successful. Note, that the update of the LDM with the individual fusion results from OR and SR as well as all non-fused data (ego positioning, CAN-sourced data from gateway, etc.) is called asynchronously by the associated threads. Since there is significant waiting time for the framework process until a database update returns successfully, the fusion process is not blocked and can consume the waiting time. The time needed by the framework to pass the various data from the network socket to the shared memory and from there to the different thread s processing functions (DAQ, OR, SR) can be neglected (few ms) compared to the delay generated by the database updates (about ms). However, since the total delay including typical but constant value database updates excluding significant data fusion processing (no OR s object fusion and SR s lane assignment, trajectory, manoeuvre, traffic estimation, etc. in these test cases) already sums up to about 50ms, it is obvious that the originally targeted cycle time of 80ms (12.5Hz) can not be hold under all environmental circumstances, i.e for many objects to be fused, predicted and updated in the LDM. This holds at least for the PG-LDM (Postgres database). Delays for NT-LDM updates (SQLite database) are not known at the time of writing and this is why the cycle time has been downsampled from the positioning cycle of 80ms to 160ms Functional Integration Testing Volvo VANET integration There is an informal report called VANET Installation - Lessons Learned available on BSCW [Ref: that describes issues on installation of the VANET router on the ebox (the hardware used in the CVIS project) using the installation package of the VANTER router version 0.8 that was ported by QFree to the CVIS platform. The router works in basic mode using the default settings for beaconing. CRF The routers have been installed on both CRF vehicles. Since the D1.5.1, the parameters to satisfy have been defined and are reported below: Verification of the automatic startup of the system; Verification of the synchronization between the router and the server on the positioning pc; Verification of the connection with the LDM; SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 19 of 72

20 Verification of the connectivity toward the other computer on the SAFESPOT net (ping to the addresses x:1:2:4:5:6) Verification of the wireless connectivity: o test that the card driver is loaded and active and that the card is correctly configured; Verification that the application sntrouter is running; Check about the LDM version; Check about the connectivity with the LDM; Check of the beacon message: base format of the message; The verification that each of the above step is successfully passed by each router is the minimum condition to consider the component as properly working. Moreover the check about the correctness presence of data within the beacon message is performed; here below an example of beacon recorded from the router is reported. headerversion : 26 sequencenumber: 67 messagetype : 1 nodetype : 1 nodeid : 77 Timestamp : ms: 970 Pos Long/Lat : 0 / 0 PosVar aa/bb : 0 / 0 Speed : 0 Heading : 0 txpower : 3 Power/Antenna : 0/3 Priority : 0 +Vehicle PAYLOAD: messageid : 0 protocolver : 25 elevation : 1000 type : 1 / 4 size WLH : 4.25 / 6.32 / 1.1 longaccel : 0 yawrate : 0 extlights : 66 accelctrl : 27 IcoMatrix : 0/0/0/0/0/0/0/0/0/0/0/0/0/ +DataElementPayload: [1]: PositionTimeStamp : 0 / 0 -SatelliteData: EMPTY Volvo Positioning integration The positioning Framework has been installed into the Volvo test truck including calibration of the camera. The Positioning Framework has only been tested in basic mode using GPS data and vehicle gateway data. The framework has provided positioning information to the system as expected. Driving in tunnel has shown that the system has degraded performance (as can be expected). This degradation has not been quantified. CRF The Positioning application has been installed in both CRF vehicles. The application is started on each vehicle with specific configuration files, that should indicate the presence of sensors on the vehicle. This is necessary to configure the application so SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 20 of 72

21 that it can use the laser scanner data on the vehicle; on the other hand this function is never enabled on the SCOVA vehicle. About the other sensor activated, the camera and the gateway are present on both CRF vehicles. Volvo Synchronization NTP The time synchronization in SafeSpot is based on the standard system for time synchronisation in Internet based systems: NTP (Network Time Protocol). The installation issues have been described in an unofficial report called How-to install and synchronize time using NTP [Ref: that was compiled after a successful installation of the NTP in the Volvo test systems. The document contains descriptions on how to install, configure and use the time NTP time synchronization in the vehicles. The performance has not been studied in detail but has been verified by using the NTP status log files. A general observation is the time is set by the positing program in such a way that the base time in the Positing PC never reaches a stable state but varies approximately +-10ms over time. The time in the other units in the system is much more stable due to the fact that the NTP system only adjusts the time gradually. CRF As already indicated by Volvo, the time synchronisation in SAFESPOT is based on the NTP systems. CRF has installed the NTP application on each pc on the vehicles: the server on the Positioning PC ( ) and the client on all the others. To allow a faster synchronisation, each pc has been left under coverage, with the NTP system running, for a around 10 hours: in the intention, this procedure should define a drift value (stored in the file ntp.drift) typical for each pc. The performance has not been studied in detail, but it has been verified directly that once time synchronised with the GPS, the two positioning pc of the vehicles: and SCOVA can differ of 1ms each other. All the other pc can be around 1ms up to 10ms of difference Shutdown sequence The shut down procedure is an essential topic that must be implemented on every SAFESPOT vehicle. The importance of this component relies on the possibility to preserve damages to the different installed pc, providing a general shut down command. The sequence is initiated by the gateway because it represents the interface with the in- vehicle network and it has the possibility to check the status of the engine. The gateway is delegate to send out the shut down message on the LAN network of the vehicle and all the SAFESPOT pc must be able to receive it, process it and provide consequent action so to avoid damages to their components due to sudden cut off of the power supply. The shut down procedure has been tested on the and SCOVA vehicles; the results are identical for the two vehicles and are reported below: SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 21 of 72

22 Gateway It starts sending the shut down message 30 seconds after the engine is off. It continues sending the same message for 120 seconds at a frequency of 1 Hz. After 120 seconds the gateway switches off. Main PC It is able to detect and process the shut down message sent by the gateway and proceed with the shut down. Positioning PC It is able to detect and process the shut down message sent by the gateway and proceed with the shut down. Laser scanner PC The shut down procedure is not implemented. IBEO has declared that the abrupt cut off of the power supply does not create any problem to their control system PC Application PC The shut down procedure has not been implemented yet. Router It is able to detect and process the shut down message sent by the gateway and proceed with the shut down. More details about the shut down procedure can be found within the deliverable D SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 22 of 72

23 4. In-vehicle validation 4.1. Functional component testing Volvo Gateways The functionality of the gateway has been tested in detail. The tests have been performed with SAFESPOT equipped vehicles. External reference sources have been used when applicable for verification. When verifying the output signals from the gateway, UDP packages have been recorded along with LDM values to compare expected value with real value sent from gateway and presented in the LDM. Signals as vehicle speed and yaw rate has required specific test cases. The expected values has been measured with an external reference source or calculated theoretically. The correlation between gateway output, LDM values and expected values has been adequate. Boolean signals have been tested standing still with engine running, activating the signals manually when possible. If not, CAN signals have been generated from an external computer. The result was not fully satisfactory. BrakePedalStatus and DoorStatus were not sent from gateway. An additional test case has been made when the Ego-vehicle is driving along a test route in real traffic. The objective was to further verify the correlation between values from Ego-vehicle with output from gateway and values stored in LDM. The result was as expected with above mentioned exceptions. CRF Concerning the testing procedure adopted to perform the validation of the component against the SAFESPOT and the SP7 specifications, after the completion of the development stage (in the very early phases of the SP1 activities) the single procedure and methodology available at the time was to follow a rigid compliancy check, parameter by parameter, against the specification, trying to gather the evidence of a reliable and compliant work both in the short term (few seconds of operation) to the long term (depending from the parameters, from ten minutes up to many hours of continuous and stable activity). Specific (and multiple) data logging and analysis tools have been developed in the (many) situations where the professional equipment of data logging available in the CRF laboratories was still not sufficient to gather the evidence of a perfect compliance with the specs. In order to give a precise example of the analysis performed in order to validate the CRF gateway, let s take the following example, where the speed and the yaw rate signals are analysed. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 23 of 72

24 Figure 11: Timing test on Gateway For both of the above pictures, the X axis represents the time, in units of 40 ms, so the whole test was carried out for something like 3.5 * 10 4 * 40 milliseconds around 20 minutes. In the Y axis, it was plotted the measured relative time of delivery of the speed (or yaw rate) UDP datagrams. The specification requires that this time is of 40 ms. In order to check for the rigid compliance of this value, the above plots were generated. As it can be observed, in average the specification is respected (most of the datagrams are delivered about 40 ms. after each other. However it can be noticed that for some singular events, this time is disobeyed, arriving up to 200 ms. or more. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 24 of 72

25 This sort of analysis (and the interventions needed to find radical solutions for this sort of issues) that has been done is very demanding. In the specific case, in order to solve completely the issue, it was necessary to intervene at the level of the CAN bus driver, by changing and reworking on it during some very intense programming sessions carried out by the most skilled specialists on the subject. After these operations, the full compliance was achieved in the long runs. We are actually not aware of further issues on the matter. The following pictures show the achieved timing characteristics for the speed and the yaw rate signals. Figure 12: Gateway repetition time rate Figure 13: Gateway repetition time rate As it can be seen, currently the repetition rate of the UDP datagrams is much closer to the nominal value of 40 ms. The residual error that can be observed (of +/- 1 ms. respect to the reference of 40 ms.) is due to relative jitter between the clocks of the signal source (the gateway) and the measuring system (the software of data logging, running on a different PC), together with the resolution of the timestamp values in the SAFESPOT system (and in the UDP datagrams produced by the gateway), which is exactly of 1 ms. The same type of analysis was performed for all of the parameters in the specifications. Further measurements of performance and the characterisation of the data throughput generated by the CRF gateway were also produced. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 25 of 72

26 The following pictures show the data transfer in bytes (Y axis) for the whole gateway system and for the two separate sources of data (the FJ radar and the CAN-B + CAN-C throughputs). The X axis is in seconds, for a total time of 1380 seconds 23 minutes of test. As it can be seen, the CAN-B + CANC throughput is very stable around 2000 bytes/s, with a negligible variance. On the contrary, the throughput of the FJ radar is affected by a significant variance, ranging from 0 to a maximum of around 2500 bytes/s. CRF POSITIONING Each positioning application release has been tested on CRF test track to verify its performance. In particular the following parameters have been checked to verify the correct integration of the module within the entire vehicle systems: The positioning software must start automatically and without manual intervention; The time and the date of the positioning pc must be synchronized by the GPS; The time stamp inserted within the UDP messages is coherent with the UTC time; The post processing on the position data of the vehicle should be aligned with the scenario performed during the test; The trajectory plot should not present any spikes or jumps, compared with the trajectories performed by the vehicles during the scenarios; The time stamp must be regularly updated every 80 ms; The trend of the absolute value of the speed must not show spikes or jumps, but it must compliant to the performed scenario; The heading angle (direction of speed) should not show any spikes or jumps and should be aligned with the map of the test track; The positioning SW must be able to process the shut down UDP message. No specific test on the performance of the different positioning modules and options has been performed. It has been noticed however a marked drift of the trajectories, provided by the positioning module, while the vehicle run a curve. The Figure 14, from the Esposytor application, reports this behavior. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 26 of 72

27 Figure 14: drift of the vehicle while driving along a curve The visible drift from the lane is as higher as the curve is sharper, moreover the positioning module is able to match again the position of the vehicle on the correct lane, after a certain distance along the CRF test track (~200 meters). CRF VANET The VANET modules have been installed on both CRF vehicles: and SCOVA. Each vehicle was equipped with a dedicated low-loss cable connected to a 5.9 GHz antenna on the roof. The vehicles, during the test have been placed in a stationary condition, one beneath the other so to be sure about the range of connectivity, but in a second moment the test have been repeated with the two vehicles starting from a high distance (~500 meters) and running in opposite directions at increasing speeds. The test performed on the VANET modules, in the conditions described above, were mainly: Test of the automatic start up of the application; Test of connectivity between the routers; Verification of the presence of the VANET messages on the SAFESPOT on vehicle LAN; SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 27 of 72

28 Verification of the correct presence of the data within the messages sent by the VANET; Verification of the respect of the time constrains; Implementation of the shut down procedure Laserscanner, CPDF IBEO has performed the functional test on the laser scanner. The results are reported in the test form in the annex LRR The LRR is installed on both the CRF vehicles. The first step to verify the correct installation is the verification of the pointing of the radar sensor on the three axes. While two of the axes can be verified with a spirit level, the pointing of the radar sensor in azimuth along the vehicle axis can be verified only on the road. In order to execute this test, the vehicle is conducted on a straight road and obstacle data sent from the radar sensor on the CAN bus are displayed as a map with a specific program developed by CRF. If the radar is pointing correctly, the front obstacle detected on a straight road should be positioned along the centre axis of the map shown on the display, as shown in the following figure. Figure 15: Object distribution in front of the radar during the functional test. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 28 of 72

29 In Figure 15, the centre yellow square indicates the front vehicle that is correctly located along the centre line. The other detected objects correspond to reflexions on the guard-rail on one side of the road. If the alignment is not correct, it is adjusted by acting on the screws of the mechanical fixation system. Once the radar sensor is pointing correctly, its detection performances can be verified by locating fixed obstacles in known positions in front of the car and analysing distances and angles measured by the sensor. Moreover, it is also possible to measure the field of view and operative range of the system by looking at what distances and angles the target is no longer detected by the sensor. By performing these tests with different positions of the target the following features have been extracted as main detection performances of the Long Range Radar. Detection performance Measured value Max operating range (car) 180 m Minimum operating range (car) 2 m Max operating range (pedestrian) 120 m Minimum operating range (pedestrian) 4 m Range accuracy (car) Max (0.5 m; 0.6%) Horizontal field of view (azimuth) ± 7.5 Vertical field of view 7.2º Azimuth accuracy 0.2º Table 2: LRR performances OR The test performed on the OR module are reported in the test form attached to this document. For more detail about the performance, please see deliverable D SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 29 of 72

30 5. Conclusions The main objective of the D1.5.2 was the description and the verification about the integration level of the different modules with the SP1 platform. This document describes the test cases and the procedures applied to verify the minimal configuration and performances of the SAFEPROVE components as the gateway, the VANET router, the Positioning PC, the laser scanner. The test cases have been designed considering the principles defined by the V schema: starting from the definition of the scenarios use case, down to the requirements specification and the system design. The verification process, then, has taken into account the input coming from the first designing phase to define the procedures and the necessary steps to have a complete module verification. This part of the validation process is described within this deliverable, where the integration of the general SAFESPOT modules represents the preliminary step to allow a complete system platform validation. It is important to underline however the integration and full compliance of the external modules is not the only condition to have a functioning and performing platform. The single module: positioning, laser scanner, gateway, must work in a coordinate and synchronized way so to provide the correct input to the SP1 components: DAQ, OR, SR up to the LDM. On the other hand, it was not a specific task for SP1 SP to proceed with a performance verification of other component provided by the other SP, and that is the reason because general procedure have been identified, focusing only on some aspects considered the minimum condition to have a working SP1 platform. More detail about the performance of the SP1 platform can be found in the D1.5.3 that should be considered as a complementary deliverable with respect to this one. SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 30 of 72

31 6. References [1] SafeSpot D1.2.3 Requirements for the Vehicle Probe Platform [2] SafeSpot D1.3.1 Data Fusion Specifications [3] SafeSpot D1.3.2 Hardware and Software Platform Specifications [4] SafeSpot D1.4.2 HW and SW specifications of prototype and test bed components, v1.3, 04/02/09 [5] SafeSpot D1.5.1 Test plan design [6] SafeSpot D1.5.3 Performance Analysis SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 31 of 72

32 7. Annexes 7.1. List of test cases (completed test forms including results) Test cases from ICCS TEST FORM SP SP1 WP WP5 Prototype Version SR module v8.2 System Configuration FW v8.2, LDM schema v9.1.6, LDM API v1.4.9 HW Release SW Release Compiled by / Company ICCS Date 09/03/09 Form Progressive Numb Functional Component Situation Refinement Reference Document D1.2.3, D1.3.1, D1.5.1 Test Type Test Objective Test Purpose Test Environment Unitary Verification Correctness In Vehicle Test Case 1: SR correctness Test Description Ego-vehicle (V1) is driving along a road. Ahead of V1 there is another moving vehicle (V2). The two vehicles are exchanging VANET beaconing messages. Data players should feed the platform and as a consequence the SR module with simulated data representing the scenario described above. Use the debug mode of framework and the logging files to check the shared memory content (IN and OUT of SR). Synchronised data from Positioning PC, OEM GW & VANET router is needed. Expected Values / Results Using the logging functions of the framework check that the logging files contain the expected information (e.g. a trajectory attached to each vehicle). The SR module works properly in the data fusion chain: DAQ-OR-SR-IP. Obtained Values / Results At the end due to a lack of synchronized players, the testing was performed on CRF Orbassano track with real vehicles. One and one SCOVA vehicle were available. The data was logged during the test and analysis was performed in the lab. The SR module was working properly in the data fusion chain in the vehicles and provided the SR_In.txt and SR_Out.txt files from the logging capabilities of the FW. The content of these txt files was including the expected input and output information for SR (e.g. trajectories). More details can be found in chapter of the current document. Status Test Case 1 OK Link to external annexed documentation (if any) SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 32 of 72

33 TEST FORM SP SP1 WP WP5 Prototype Version SR module v8.2 System Configuration FW v8.2, LDM schema v9.1.6, LDM API v1.4.9 HW Release SW Release Compiled by / Company ICCS Date 10/03/09 Form Progressive Numb Functional Component Situation Refinement Reference Document D1.2.3, D1.3.1, D1.5.1 Test Type Test Objective Test Purpose Test Environment Unitary Validation Performance In Vehicle Test Case 2: SR timing performance Test Description Ego-vehicle (V1) is driving along a road. Ahead of V1 there is another moving vehicle (V2). The two vehicles are exchanging VANET beaconing messages. Data players should feed the platform and as a consequence the SR module with simulated data representing the scenario described above. Measure processing time of SR module. Use e.g. QueryPerformanceCounter in Windows for measuring time. Synchronised data from Positioning PC, OEM GW & VANET router is needed. Also static map data should be available. Expected Values / Results The measured SR processing time should be compliant to the timing requirements. Obtained Values / Results Also in this test case due to lack of synchronized players, the testing was performed on CRF Orbassano track with real vehicles. One and one SCOVA vehicle were used. The data was logged during the test and analysis was performed in the lab. The exact SAFESPOT configuration (main PC-eBox, positioning PC, application PC, etc.) inside CRF vehicles was used. The following results were obtained concerning the average, min and max duration of one SR cycle inside the vehicle. Number of cycles: 646 Average duration of SR: 12,69 ms Max observed duration: 49 ms Min observed duration: 8 ms More details can be found in chapter 7 of D Status Test Case 2 OK Link to external annexed documentation (if any) SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 33 of 72

34 Test Case 3: Maneuver detection and object to lane assignment Test Description Ego-vehicle (V1) is driving along a road. Ahead of V1 there is another moving vehicle (V2). The two vehicles are exchanging VANET beaconing messages. The two vehicles are getting closer and moving apart, they perform maneuvers (e.g. lane changes, overtaking). Data players should feed the SR module with simulated data representing the maneuvers performed. Synchronised data from Positioning PC, OEM GW & VANET router is needed. Also static map data should be available. A visualization of the scenarion with the use of the SafeFusion simulator will help the overall validation. Expected Values / Results Each vehicle should be assigned a correct lane. A maneuver should be attached to V1. This test is highly dependent on the positioning accuracy and lane geometry availability. Obtained Values / Results Due to lack of a synchronized player at the time of writing, real scenarios with the two CRF equipped vehicles took pace at the Orbassano track. For checking the assignment of a vehicle to a lane module, two scenarios were performed. The only vehicle that participated was the equipped one. This vehicle performed the following two tests: 1 st scenario: vehicle driving on the right lane 2 nd scenario: vehicle driving on the left lane The results showed good performance of object to lane assignment algorithm. For more details see chapter 7 of D For checking the maneuver detection algorithm, an overtaking maneuver was performed on the straight segment of the Orbassano track. The maneuver was performed by the vehicle, which initially followed the SCOVA one. The overtaking maneuver was correctly detected by the system. More details in chapter 7 of D Status Test Case 3 OK Link to external annexed documentation (if any) SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 34 of 72

35 Test Case 4: Trajectory estimation Test Description Ego-vehicle (V1) is driving along a road. Ahead of V1 there is another moving vehicle (V2). The two vehicles are exchanging VANET beaconing messages. Data players should feed the SR module with simulated data representing the scenario. Synchronised data from Positioning PC, OEM GW & VANET router is needed. A visualization of the scenarion with the use of the SafeFusion simulator will help the overall validation. Expected Values / Results A trajectory should be attached to each vehicle. The correctness of these predicted trajectories will be checked. Obtained Values / Results Again in this scenario real data were used (as above). The two CRF equipped vehicles were driving on Orbassano track. A trajectory was attached to each vehicle. A quick first visual validation of the predicted trajectories can be seen in the Safefusion simulator. A screenshot of SafeFusion simulator depicting the trajectories of both vehicles can be seen below: As ground truth data for the trajectory validation the future positions from the SP3-Positioning were used. The following results are from one lap on CRF test track in Orbassano: Average distance of future points from trajectory: 0, m Standard deviation: 0, m More details can be found in chapter 7 of D Status Test Case 4 OK Link to external annexed documentation (if any) Test Case 5: Traffic estimation SF_D1.5.2_InVehiclePlatformTestResults_v1.2.doc - Page 35 of 72