Camera Behavior Models for ADAS and AD functions with Open Simulation Interface and Functional Mockup Interface

Transcription

1 Camera Behavior Models for ADAS and AD functions with Open Simulation Interface and Functional Mockup Interface Kmeid Saad Stefan-Alexander Schneider Master s Course of Advanced Driver Assistance Systems, University of Applied Sciences Kempten, Germany, {kmeid.saad, stefan-alexander.schneider}@hs-kempten.de Abstract Advanced driver assistance systems (ADAS) and Autonomous Driving (AD) provide among comfort to the driver also a great potential for future mobility and tends to increase traffic and car safety. All ADAS and AD functions and especially those associated with high safety levels, require paradigm-changing approaches for the homologation: some ADAS and AD functions require up to about 200 million km [1] of real drive testing for the qualification. This amount of real drive testing is not feasible for any OEM and therefore there is a strong need for a hybrid test strategy where the performed real drive tests take credit from a virtual campaign evaluation. Such a combined real and virtual test strategy could reduce the necessary efforts for the qualification of a given function development and its validation. This homologation method is not new and already applied at the homologation of vehicle dynamic driving functions, like the Electronically Stability Program (ESP) [2]. To support virtual testing, the triangle of the driver, the vehicle and the environment has to be modeled for simulation. The interface between the vehicle and the environment, i.e. a sensor that is a device to transform physical information into electrical signals, is of crucial importance for the ADAS and AD function, because it replaces step by step the function driver in the car. This paper focuses on sensor behavioral models, specifically camera behavioral model, as a part of a new tool chain method that combines integrations platforms and authoring tools by the Open Simulation Interface [8] and the Functional Mockup Interface [9]. The integration is following the concept of an integration of a Functional Mockup Unit (FMU) with a specific add-on for the environment semantics by OSI. Such a corresponding container is a called in the following Open Simulation Functional Mockup Unit (OFU). An OFU with, e.g., a camera behavior model will be able to provide a simulation model with more realistic Video Data Stream (VDS) for image based ADAS and AD functions and the necessary Vehicle Detection (VD). This paper also provides an outlook on how to use this basic architecture. Keywords: OSI, FMI, OFU, virtual testing, camera behavior model, image based ADAS and AD functions. 1 Introduction Reliable ADAS and AD functions are of crucial importance today. However, in the presence of so many homologation requirements to be fulfilled and as the vehicle s complexity is constantly increasing, the validation of ADAS and AD functions is getting more and more time consuming and very expensive. No wonder that the automotive suppliers are forced to deliver sensor behavior models to the automotive OEMs. The testing effort often consume more than 50% of the overall development effort [2]. Late fault and failure detection normally leads to huge corrections and additionally maintenance costs. Reducing quality or compromising on functional safety for ADAS and AD functions is often not acceptable. Therefore, the main problem remains how to improve test coverage for a prescribed product s functionality and reliability. One way to overcome this challenge is to introduce and/or to increase the amount of Virtual Testing (VT) in ADAS or AD function development. This results in a hybrid test strategy for real and virtual testing. VT methods replaces real drive test by virtual tests of the ADAS or AD function; instead of the real car to be tested itself. In VT, functions can be characterized, and their performance can also be predicted via the usage of simulation models. Therefore, VT can also support ADAS and AD function s validation and function development as well as the homologation, where in a qualification process the virtual tests can provide support and integrate if not replace the actual car test as well as the actual car component. By verifying the adequacy of the modelling results obtained from VT and comparing them to real drive behavior, it is argued that this will lead to savings in both development resources as well as in development time by partially or totally depending on result of the corresponding VT [4]. 1

2 In some case modeling ADAS sensors can turn out to be a very challenging task, especially when considering various but necessary modeling approaches needed to replicate the actual sensor s behavior as close and reliably as possible. When we look, e.g. at the most commonly used ADAS and AD sensors, we can definitely see some of the common and specific factors that affect the sensor s behavior. 1. Camera, Radar and Lidar are affected by: a. Target distance, b. Field of View, and c. Housing, mounting position. 2. Camera: a. Visibility due to Weather situation, and b. Night/Day time. 3. Radar: a. Antenna Diagrams, b. Ducting/Echoes, and c. Resolution(angle/distance/Speed). 4. Lidar: a. Number of beams, angular resolution, b. Object characteristics, material composition, and c. Noise resistance. These factors are not only necessary to be provided as a valid input to our sensor models, but also in the most standardized and interchangeable way; a perfect justification for OFU. The Functional Mockup Interface (FMI) supports to develop products controlled by a complex set of physical laws that can be represented by virtual products assembled from a set of digital models and control systems. Each set represents a combination of parts capable of simulating real product s functionality [5]. The FMI standard thus provides the means for model based development of systems, e.g. it can be used in designing ADAS and AD functions that are driven by electronic devices inside vehicles (e.g. ESP controllers, multi-functional cameras, radars). Activities from systems modelling, simulation, validation and test can be covered with the FMI based approach. The open simulation interface (OSI) contains an object based environment description using the message format of the protocol buffers library developed and maintained by Google [6]. OSI consists of two individual top level messages defining the so called Ground Truth Interface (GTI) and the Sensor Data Interface (SDI). The GTI gives an exact view on the simulated objects in a global coordinate system. The SDI describes the objects in the reference frame of a sensor for environmental perception [7]. A Multi-Functional Camera (MFC), e.g., is a specific ADAS and AD product capable of implementing numerous functions like vehicle detection, lane departure warning, traffic sign detection,, thus making the everyday driving safer. The contribution of new ADAS and AD functions regarding accident prevention is enhancing. According to the NHTSA in the US a statistical projection of traffic fatalities for the first half of 2016 shows that an estimated 17,775 people died in motor vehicle traffic crashes (NHTSA, 2016). A lot of those accidents could have been prevented by correct recognition of valid targets for instance with VD. VD function is constantly scanning the field of view (FOV) in front of the ego car and trying to find valid targets, i.e. vehicles. Taking in consideration that some VD method could be highly depend on specific pattern aspect ratios or features and could also be estimating the distance to the vehicle using pixel distance difference between a detected vehicle and the hood, it is clear how a more realistic VDS can be of an essence for VD s virtual testing approach. Further in this paper, we will demonstrate how the VT approach used for environment simulation and FMI approach used for physical camera model and ADAS and AD function integration can be combined to aid in the process of developing, testing and even validation of ADAS and AD functions (VD). 2 Specific Aims A series of studies is proposed to investigate the usage of VT approach and FMI approach in testing and validating new developed ADAS and AD function coupled with sensor models, i.e. the physical camera behavioral model, VD in our case. First, we intend to study the virtual testing approach by selecting an appropriate integration platform capable of implementing OFUs. The integration platform should be capable of interacting with the implemented OFU providing it with all necessary inputs coming from the virtual platform, and be able to receive necessary outputs from the OFU. Second, we shall study the possibilities of creating an OFU based on the FMI capable of integrating a camera behavior model defining all of its necessary inputs from outside environment (for both real and virtual) and its specific outputs for function testing and validation. 2

3 Another OFU representing the VD function will be created. The VD-OFU should be able to interact with the camera behavior model-ofu. The VDS provided by the camera behavior model-ofu is meant to reflect the behavior of a realistic camera to a certain percentage by implementing relevant image distortions and aberrations on the ideal VDS. The manipulated VDS will then be the main input for the VD-OFU and not the ideal VDS generated by the virtual simulation/integration platform. 2.1 VT Approach The aim of this approach is to investigate the capabilities of an integration platform in reflecting specific environmental details in a digital/virtualized format. This virtualized data should respect a certain level of abstraction necessary for various function simulation and validation. The integration platform must have the capabilities of integrating OFU components into its environment, more specifically OFU components representing a camera behavior model and the VD function. In addition to its capabilities of integrating OFUs, we should be able to generate various test scenarios reflecting possible real life situations. Integration platforms should also provide a certain level or reliability so that all test results can be used further on for early design modification and for satisfying various homologation requirements. 2.2 FMI Approach Based on the FMI standard, various OFU units can interact with each other and with the integration platform easily. The camera behavior model-ofu and the VD-OFU will be integrated in the same integration platform sharing inputs and outputs, as shown in the following figure. Integration/Simulation Plattform Ideal VDS OSI Data OFU 1 Physical Camera Behavioral Model Manipulated VDS OSI Data OFU 2 VD Figure 1. Example for the basic architecture of OFUs. 2.3 Vehicle Detection (VD) function (machine learning based) Vehicle Object List (OSI) The aim of the VD function is to detect vehicles inside the FOV of the imaging device and estimate the distance between them and the ego car. By using standard computer vision algorithms and machine learning algorithms it s possible to analyze a camera s video data and find signs of vehicles. This can be done by following algorithm: 1) Extract training features for vehicle and nonvehicle objects (could be done offline), 2) Train the model using linear support vector classification (could be done offline), 3) Implement a sliding window approach to scan each frame and 4) Use the extracted windows and the trained model to generate an object list (OL) with the detected vehicles. 3 Research and Design Methods In this section, we present in details the research design and methodology for the proposed approaches. For the VT approach, we used CarMaker from IPG as an integration platform whereas for the FMI approach we describe the development of the camera behavior model and a VD-Function into separate OFU units and how they are integrated in CarMaker. 3.1 CarMaker in VT Approach An integration platform (IP) is a development environment, e.g. a software tool, that enables to integrate various functions like the driver, the vehicle or the sensor behavior that describe in total a system in its use like a car on the motorway. IPs are the most important element in the VT approach. CarMaker is an example of an IP in today s market. It represents an open integration and test platform and enables a wide spectrum of applications including the classic vehicle dynamics simulation. In CarMaker, the virtual vehicle contains almost all parts of a real vehicle, including powertrain, tires, brakes, etc. It is also possible to integrate real automotive controllers, e.g. ABS, ESP, ACC, or software modeled controllers. The basic test scenario created in CarMaker includes the configuration of: 1) Demo car (integrating OFUs), 2) Test track, 3) Driving maneuver (speed, acceleration, braking force, etc ) and 4) Environmental factors (day, night, rain, fog, position of the sun, ). 3

4 3.2 Camera behavior model as an OFU VDS is mainly obtained from CarMaker by positioning a virtual camera in the ego car defined by the mounting position. Initial VDS generated by the virtual camera represents an ideal image of the artificial environment included in the camera s FOV. The main purpose of the camera model-ofu is to be able to handle the original VDS and apply on it image transformation algorithms, thus generating a more realistic video data stream (VDS-OFU). At this point, it is important to point out that although all image processing techniques will be directly implemented inside the OFU, the ideal VDS will be provided by a TCP or a UDP connection between the OFU and the integration platform CarMaker, whereas all other ground truth data, vehicle dynamics and other sensor data (if needed) will be provided via the standardized OSI buffer. Two main technical steps of a camera are modeled by the corresponding OFU, i.e. the so-called Optical acquisition and the Image acquisition : 1. Optical acquisition represents the first contact of the camera with the analog environment where a system of lenses collects and focalizes light in order to be projected on an active sensor grid. 2. Image acquisition represents the second contact of the camera with the analog environment, which is already provided via Optical acquisition as a focalized and concentrated light waves over the active sensor grid. At this stage the analog signal is transformed to a digital signal corresponding to its intensity. VDS-OFU reflects the relevant image distortions and aberrations applied to the ideal VDS. In our current demonstration, we will focus on the modeling of the following effects (similarly additionally effects can also be implemented as long as they are defined by algorithms using VDS and OSI): Image Distortions Here, we classify distortion into radial and tangential distortion, where: 1. Radial distortion occurs when light rays bend more near the edges of a lens than they do at its optical center. Figure 2. Radial distortions (see mathworks.com). 2. Tangential distortion occurs when the lens and the image plane are not parallel. Figure 3. Tangential distortions (see mathworks.com) Lens Blur In an ideal situation, each small point within the object would be represented by a small, well-defined point within the image. In reality, the "image" of each object point is spread, or blurred, within the image. this places a definite limit on the amount of detail (object smallness) that can be visualized. Figure 4.a Ideal Image. Figure 4.b Blurred Image (taken with the camera) Lens Flare Lens flare is an unintended effect caused by rays passing through the camera lens in an unintended way. It is due to inter-reflections of lens elements of a compound lens and diffraction, where it ads various artifacts to photos, like multiple ghosts. 4

5 3.3.1 VDS-Input To be able to detect e.g. the vehicles, a multifunctional camera should be analyzing the situation upfront. The video stream provided by camera behavior model (VDS-OFU) represents the main input for the VD-OFU detection. Figure 5. Multiple ghosts due to lens flare (taken with a camera) Vignetting The effect of the vignetting may be described as the reduction of an image's brightness the periphery compared to the image center. Vignetting is caused by camera settings or lens limitations. Figure 6. Vignetting image from an integration sphere (taken with a camera). 3.3 VD Function as an OFU By integrating the VD function in an OFU, it is possible to transfer it to several systems and integration platforms, e.g. it can also be tested with a real camera, when the input is mapped to the VD- OFU. From a black box perspective, VD s functionality can be separated into three essential parts: VDSinput, VD-process, and OL-output VD-Processing Those frames are analyzed by the VD function as mentioned above. Computer vision libraries are used for the image analysis. The VD algorithm is capable of detecting vehicles at varying distances from the ego car and with different color and dimension properties. The VD is also capable of estimating the distance between the ego car and the detected vehicle OL-Output VD function outputs an OL with the number of detected vehicle, estimated distances to targets and possible inlane indicating whether the detected vehicle is in the ego car s driving lane or not. Inattentive drivers could be warned from a sudden emerging dangerous situation. It s also imaginable to support an emergency brake assistant by initiating an emergency braking when distance to the next car is critical. Provided with a training data, Histogram of Oriented Gradient (HOG) features were extracted and then used to train a support vector classifier. Later a sliding windows approach is implemented where overlapping tiles from each frame are then classified as vehicle or non-vehicle. Finally, heat maps to show locations of repeated detections helped in identifying detections that were found in the same location or near the same location in several subsequent frames, thus minimizing the false positive rate. A similar approach was also used to generate bounding boxes around high-confidence detections where multiple overlapping detections occur and to estimate the distance between the detected vehicles and the ego car. VD algorithm is described in the following flow chart. Input VDS Processing Vehicle Detection (OFU) Object List Figure 7. VDS-Input, VD-Processing, and OL-Output. 5

6 Training: Training Set: 1)Vehicle 2)Non-Vehicle Processing (VD): Frame: From Camera Model Feature Extraction: 1)Color Features 2)Gradient based Features Train a Classifier: 1)Linear SVC Search for Vehicles: 1)Sliding Window technique Feature Extraction: 1)Color Features 2)Gradient based Features Classifier: 1)Linear SVC Reduce False Positives: 1)Heat map technique Figure 10. Distorted frame, output of sensor model. By closely examining figure 9 and 10, we can see the effects of radial and tangential distortions where a clear shift in the pixels positions is taking place. It is important to point out that this shift was not arbitrary generated, but by the actual computation of the camera s intrinsic calibration matrix. Object List: 1)Bounding Boxes 2)Distance estimation Figure 8. Flow chart for VD. 4 Preliminary Results Taking in consideration, the current state of the research with a lot of potential in this section we will present the first results we obtained on our camera physical modeling approach, OFU performing edge detection and our machine based VD algorithm Camera Physical Behavioral Model Figure 11. Pixel difference, output of sensor model. Figure 11 shows a maximum diagonal shit of pixels this indicates that some objects in the camera s FOV may appear to be closer to the camera s center of projection by pixels. Figure 9. Ideal frame showing no distortion. Figure 12. Ideal frame to the left, blurred frame to the right (output of camera model). 6

7 Another important camera feature, is its ability to resolve a necessary level of details. Lens train properties and minor imperfections sum up in some cases to produce an undesired blurry image. In figure 12 we show how after computing our camera s blur kernel we were able to replicate this phenomena with our camera model, thus obtaining a more realistic and more similar to the real camera s image. In figure 15 we show our initial attempt to model the GHOST effect. The goal is to show that such GHOST artifacts are very important to study and reproduce in camera models because when judging by their diameter, which is constant, we can see how this effect can totally obscure a vehicle, or any other traffic participant inside the camera s FOV. This is demonstrated in figures 15.b and 15.c. Figure 13. Ideal frame (solid background). In figure 13 and 14 we demonstrate how would our image look like with and without the vignetting effect. Examining different levels of average pixel intensity we could definitely see how crucial it is to implement the vignetting effect especially for functions that are color / intensity based. Figure 15.b Ghosting, car-distance (10-15) m Figure 15.c Ghosting, obscured car-distance (10-15) m Figure 14. Vignetting effect, output of sensor model. 4.2 VD Algorithm In this section we show some of our results in developing the VD algorithm, it is important to emphasize that developing the VD algorithm by ourselves and thus possessing the source code and all of its specific and detailed features will be of a great advantage for us in the upcoming camera model validation process. In figure 16 and 17 we can see the histogram of oriented gradient (HOG) represented respectively for a vehicle and a none vehicle example. Figure 15.a Ghosting, result of Lens Flare (early model), the red border represents the camera s FOV. 7

8 Figure 16. Vehicle features extraction. Figure 19. Heat map and bounding boxes. 4.3 OFU VDS In this part we demonstrate the possibility of transferring a VDS to the OFU which is running an edge detection algorithm. Figure 17. Non-Vehicle features extraction. In figure 18 we can see the sliding window s approach implemented in the scope of searching for car features. The different colored boxes represent a number of overlaying searching boxes. Here it is important to point out the position of the white car, at the very right margin of the FOV exactly where distortion effects are most significant. In figure 20 we can see the IPG Movie representing a VDS of a specific CarMaker testing scenario. In this scenario the ego car is being configured with a VDS- OFMU with the main task of collecting the simulation s ground truth and sensor data (Ideal VDS in this case). Figure 20. Testing scenario in CarMaker (Ideal VDS). Figure 18. Sliding window approach. In figure 19 we can see some of the VD s obtained results where starting form the 1st coloumn the initial frames are presented then the heat map, based on multiple detections in a certain number of frames (used to decrease the probability of false positives), and finally at the last column we can see the bounded boxes arround the successfully detected vehicles. Figure 21 represents the output of an edge detection algorithm that we implemented inside the VDS- OFMU, thus demonstrating the following: 1. A successful VDS transfer from the simulation environment to the VDS-OFMU. 2. A successful computer vision algorithm implementation inside the VDS-OFMU. 3. A successful VDS output from the VDS- OFMU. 8

9 Further on with our adequate measurement interface, the previously mentioned test bed, the virtual simulation tools and our OFMUs for our sensor models and ADAS and AD functions we will be ready to extend our tool chain to validation. Figure 21. Edge detection performed and OFU output. 5 Short look into Validation As mentioned before we intend to use our tool chain not only for ADAS and AD function development and virtual testing, but also for sensor model validation and later on for ADAS and AD function validation. To do that we started to build up our first test bed prototype, to be used for camera model validation. Figure 22. Test bed prototype for camera model validation. The test bed represented in figure 22 is capable of controlling the camera s pitch, yaw and roll angles and one additional translation, all which can be easily controlled via a dedicated desktop interface as represented in figures 23. Figure 23 Dedicated test bed control interface. 6 Conclusion In this paper, we intended to show that virtual simulation is a very important pillar in ADAS and AD function development and validation. In order to successfully virtualize this process, we showed that sensor behavior models and standardized interfaces, especially OSI and FMI, are of an essence. Several challenges in design and sensor behavior model validation are still up to come however, we did manage to show the evident difference in image quality between ideal VDS and camera behavior model VDS. We also managed to show that an OFU is capable of receiving and processing VDS, thus proving that this is easily extendable from Computer Vision algorithms to machine learning based detection implementations or localization and control functions. Our up-coming tasks would be to further develop the camera sensor model and integrate it with the machine learning VD algorithm as OFUs thus performing much more complicated tasks. Future work will also be to apply this new method to other sensors not only cameras but radars, lidars, etc. and the validation of the corresponding sensor behavior models vs. the real sensor behavior. References [1] Handbook of Driver Assistance Systems, Editors: Winner, H., Hakuli, S., Lotz, F., Singer, C. [2] Elberzhager, F., Rosbach, A., Münch, J., & Eschbach, R. 4th International Conference, SWQD Inspection and Test Process Integration Based on Explicit Test Prioritization strategies, pp , [3] Keynote Speaker L. Rogowski from Continental at the IPG Open House 2015, see automotive.com/de/veranstaltungen/open-house- 2018/rueckblick [4] C. Cifaldi. Virtual Test and Engineering Simulation in Aerospace & Defense, [5] Blochwitz, M., Otter, M., Arnold, M., Bausch, C., Clauß, C., Elmqvist, H., Peetz, J.-V., Wolf, S. The Funcional Mockup Interface for Tool independent Exchange of Simulation Models, [6] [7] [8] Open Simulation Interface OSI, see 9

10 [9] Functional Mockup Interface, see 10