MARK. January 2019 issue... BENCH THE INTERNATIONAL MAGAZINE FOR ENGINEERING DESIGNERS & ANALYSTS FROM NAFEMS

Transcription

1 BENCH MARK January 2019 issue... Simulation Limited: How Sensor Simulation for Self-driving Vehicles is Limited by Game Engine Based Simulators A Guide to the Internet of Things Simulation of Complex Brain Surgery with a Personalized Brain Model Learn How to See Prediction of Clothing Pressure Distribution on the Human Body for Wearable Health Monitoring What is Uncertainty Quantification (UQ)? Efficient Preparation of Quality Simulation Models - An Event Summary Excel for Engineers and other STEM Professionals THE INTERNATIONAL MAGAZINE FOR ENGINEERING DESIGNERS & ANALYSTS FROM NAFEMS

2 Simulation Limited: How Sensor Simulation for Self-driving Vehicles is Limited by Game Engine Based Simulators Zoltán Hortsin, AImotive Simulating the images of unique cameras used for self-driving was a driving force behind the development of a new engine for a simulator designed for autonomous vehicle development. Over the last 18 months, the importance of simulation in the development of autonomous vehicle technologies has become widely accepted. Industry experts and analysts alike are claiming that enormous distances must be covered by self-driving systems to achieve safe operation in varied road conditions and environments. Meanwhile, certain weather conditions and traffic scenarios are extremely rare, resulting in limited test opportunities. The only way to cover the distances and reach the diversity required for the safe operation of highly automated vehicles is through testing these systems and their modules in virtual environments. The demands of simulation for self-driving cars are extremely high and not all simulators are created equal. Simulation for Self-driving Simulators for autonomous technologies must be comprehensive and robust tools; offering at the least: 1. A diversity of maps, environments, conditions and driving cultures. 2. Repeatability of tests and scenarios for the continuous development of safety-critical systems. 3. Ready access for self-driving developers and engineers to accelerate iteration times and development loops. However, these requirements only scratch the surface. The demands of self-driving are unique, as a result, only a purpose-built virtualization environment can become a full solution. Several industry stakeholders have built proprietary solutions based on game engines. While game engine rendered simulators can provide the above-listed characteristics, they do not solve all the challenges of self-driving simulation. Beyond the basic demands listed above a true self-driving simulator has to offer more: 1. The utmost level of physical realism in vehicle and road models, weather conditions, and sensor simulation. 2. Pixel-precise deterministic rendering to ensure that minor differences in simulated environments do not affect the results of repeated tests. 3. The efficient use of hardware resources including the ability to run on any system from laptops to servers or utilizing the performance of multiple CPUs and GPUs. The Limitations of Game Engines These more specific demands cannot be answered efficiently by game engine based simulators. Their focus is inherently different. A game engine is designed with a commercial end-user in mind and is built to utilize average hardware setups while being optimized to offer the best game performance. Furthermore, it is often 3D and graphic artists that create the final visual effect to ensure a spectacular world for the consumer, rather than a physically correct environment.

3

4 AImotive encountered several problems while using a game engine-based simulator. One of the most notable was how game engines are not prepared to simulate the images of unique sensors such as ultra-wide field of view fisheye lenses or narrow view long-range cameras. This is fundamentally because such views are almost never used in games. As a result, the previous iteration of our simulator contained custom modifications to the engine. However, as these were not organic elements of the code they were also a bottleneck. On the one hand, they affected the performance of the simulator. On the other hand, certain effects built into the game engine could not be used on intermediary images, only on the final render, which resulted in less realism. Of these more unique sensors, ultra-wide field of view fisheye cameras were the most exciting challenge. The following sections will outline the approach we took to overcome these difficulties. Synthesizing Camera Images A camera s operation can be divided into two parts: the optics and the sensor. Rays of light move through the lens and arrive in a given pixel of the sensor. When simulating camera images, it is this process that has to be recreated. There are several mathematical models for this projection, the commonality between them being that they all result in a 2D pixel position from 3D data. However, when discussing images taken with ultra-wideangle lenses, further difficulties arise. We use fisheye lenses to cover ultra-wide field of views which do not create rectilinear images, but exhibit barrel distortion. As a result, projecting them onto a two-dimensional plane is a slightly more complex process as the distortion must be accounted for. Figure 1 illustrates how this distortion is mapped to a 2D plane. Simulating a Fisheye Image The most obvious solution to simulate such an image would be to simulate the exact paths of the rays of light as they move through a simulated lens and onto the sensor. However, this approach relies on ray tracing, a technology currently only available in a few select GPUs. Most GPUs employ rasterization for image synthesizing (Figure 2), a solution that does not allow for a robust fisheye projection. While there are certain workarounds that make the projection possible, the nature of these solutions mean that the projection will not be robust, thus, rendering may be incorrect, or the performance of the engine may be affected. To find a solution the problem has to be reexamined. Reexamining the Problem To achieve a robust projection the task at hand has to be divided into two parts. In the first step, data for rays of light reaching the lens is generated. In the second step the corresponding data for the rays that reach the sensor must be found. Naturally, the high distortion of fisheye, lenses causes difficulty in the projection. Step one This is the most demanding part of the process, which relies most heavily on the rasterization performance of the GPU. Based on the calculated location of the sensor, images must be generated that cover the area of space from which light can enter the lens. This can be a single image or a total of six, depending on the characteristics of the simulated camera, as shown in Figure 3. The hardware demands of the process are heavily dependent on the quality of these images, and to achieve the highest degree of reality highquality images and physically based High Dynamic Range (HDR) rendering must be employed. As several cameras and other sensors are being simulated concurrently by the simulator, this can lead to huge demands on memory and compute capacity. Hence the ability to efficiently utilize multiple CPUs and GPUs if needed is vital. However, to ensure flexibility, the system should also be able to run on more everyday systems such as desktop PCs. This is to allow developers and self-driving testers to have proper access to the simulator and use it as a development tool. Naturally, high-quality tests have to be run on heavy performance setups, but not all tests require such resolutions. Figure 1: Fisheye projection illustrating rays and final pixels Figure 2 : GPU rasterizing

5 Simulation Limited Figure 3: Six images forming a cube with a camera placed in the center Figure 4: Pixels from images are projected onto a sphere Step two In essence, the task at hand is to simulate the lens distortion of the camera and create the image that appears on the sensor itself. The resolution of this image matches the resolution of the sensor. The data used to create the image is obtained from the 1-6 images generated in step one. The GPU is used to calculate the characteristics of the 3D rays that correspond to the 2D pixels on these images (Figure 4). Further characteristics of the simulated camera (exposure time, tonemapping etc.) are also added in this step. These are needed to create a simulated sensor image that is the closest it can be to the image the real sensor would provide to the selfdriving system in the real world. Only through a high correlation can simulation truly be a viable tool for the development and validation of self-driving systems. The Simulated Image Following these steps, we can create a virtual representation of an image that a fisheye camera would provide to the self-driving system. The robustness of this solution is based on recreating the problem in a way that allows GPUs to compute the simulation effectively without relying on niche technologies. The method is also extremely precise, allowing for pixel-perfect deterministic rendering: each scene will be calculated and rendered in entirely the same way, every time it is loaded. If simulators are to be used as a platform for not only testing but also validation, they must be the closest possible representation of the real world in every regard. The example given above clearly presents how the inherent limitations of game engines cannot serve as a reliable platform for this. Fisheye cameras are an important element of a vision-first self-driving setup as they can be used to create a wide (or ultra-wide) field of view around the vehicle easily. Without the ability to properly simulate these there will be limits to the correlation between the real world and the simulated test environment. Beyond this highly technical difficulty and the required level of physical fidelity, the limitations discussed above regarding the flexibility and scalability of the simulator, the efficient use of hardware are also important factors. Building on our experience with selfdriving technology and working with a game-enginebased simulator in the past we strongly believe that in the future simulators for autonomous technology testing and verification must be purpose built. Zoltán Hortsin is a self-trained game engine development and OpenGL ES 2, 3 and Vulkan 3D API specialist who originally studied Dentistry at Semmelweis University, Budapest. After discontinuing his university studies during his final semester, he joined AImotive's predecessor, Kishonti Informatics Ltd, where he served as a team leader to work on the company's mobile GPU benchmarks. Zoltan is the technological mentor of AImotive's internal simulation engine development team. His favorite area of research is real-time global illumination. Zoltán is a chocolate connoisseur, and spends his free time reading up on paleontology and collecting fossils. To read more about AImotive visit aimotive.com

6 Join us +44 (0) nafems.org/joinus