VISUAL NAVIGATION SYSTEM ON WINDSHIELD HEAD-UP DISPLAY

Transcription

1 VISUAL NAVIGATION SYSTEM ON WINDSHIELD HEAD-UP DISPLAY Akihiko SATO *, Itaru KITAHARA, Yoshinari KAMEDA, Yuichi OHTA Department of Intelligent Interaction Technologies, Graduate School of Systems and Information Technologies, University of Tsukuba Tennodai, Tsukuba, Ibaraki, , Japan TEL: FAX: {kitahara, kameda, ABSTRACT We introduce a unique visual navigation system for car drivers. In this system, a driver can see visual information on a windshield head-up display (WHUD), of which the display area is a whole windshield. We implemented a WHUD on a real vehicle and we realized a new navigation system that can shows direction in navigation and distance to a destination as a visual sign. A driver needs little eye movement to see them. It is easy for drivers to recognize them because the sign position for destination on the WHUD directly indicates the orientation toward the destination. A GPS and a geomagnetic sensor are used to estimate the location and the orientation of the vehicle. The position of navigation signs on the WHUD moves appropriately as the vehicle goes. KEYWORDS driver assistance, WHUD, navigation, visual support INTRODUCTION Drivers get many kinds of information from a vehicle about the status of the vehicle such as speed or fuel. And there are many kinds of driving support system that provides drivers live information like navigation information or live view of a monitoring camera on the vehicle. 1

2 Among the various kinds of information given to the drivers, visual information is useful because it is easy to understand, and so it can be recognized in short time. In addition, it can convey rich information to drivers. However, drivers have to move their eyes from the outside road scene to the inside monitor or meters in order to get the visual information. Such eye movement is sometimes unsafe especially when they are maneuvering. A head-up display (HUD) is a good device to provide visual information to a driver. A display device of HUD is usually set just in front of driver s seat near the windshield, so the driver can see the information displayed on the HUD with little eye movement and with little refocusing. By HUDs, drivers can keep watching the outside scene in front of their vehicle even when they need some information on driving. Since display area of ordinary HUDs is small, advanced HUDs that have wider display area have been developed []. The HUDs will use the whole windshield area as the display. We call it a windshield head-up display (WHUD). In this paper, we describe our preliminary implementation of WHUD on a real vehicle and we introduce a new visual navigation system that shows information of the direction and the distance to a destination on the WHUD. IMPLEMENTATION OF WINDSHIELD HEAD-UP DISPLAY Figure 1 The design of the WHUD system. The light from the projector is converged at driver s eye position. HUDs should achieve high luminance and wide display area. In daytime, driver should be able to see the displayed information on the HUD against the strong light coming from the outside. Currently most of the applications of HUD are for nighttime use. To realize a brighter HUD, we designed the WHUD system shown in figure 1. In our system, the light from the

3 projector is converged at the driver s eye position. As shown in the figure, a retroreflective material is spread on the dashboard. The projector is set on the roof of the vehicle and the light of the projector is projected to the forward direction of the vehicle. And then it is reflected by a mirror so that it goes through the windshield. The retroreflector on the dashboard returns the light to the direction it comes from. On its way back to the projector, some of the light is reflected by the windshield. The reflected light will converge on a single point that is an optically symmetric position of the projector. By arranging the position close to the driver s eye position, the driver can see the bright image from the screen together with the outside scene. Our implementation makes the installation and the adjustment relatively easy though a certain size of structure has to be installed on the top of the vehicle. It also enables the HUD to have a wide display area. The pictures of the WHUD on a vehicle are shown in Figure. Figure Overview of the WHUD system. The right picture shows the high luminance and wide display area of the WHUD. WHUD SETUP Distortion correction As seen in Figure, an image projected to the windshield is distorted because the windshield is not flat. In order to show an image with no distortion, we heed image warping process. First, we assume that distortion caused by the windshield can be modeled by warping transformation function W. If an image I is projected, the driver observes an image W (I). W is a nonlinear function. It consists of a set of parameters each of which defines the correspondence between a pixel in an original image given to the projector and a pixel observed by a camera at the driver s eye position. The transformation of inner area of the correspondence points is interpolated by affine transformation. The procedure of distortion collection is as follows. 1. Project a grid pattern image on the HUD.. Capture an image by a camera that is set at the driver s eye position. 3

4 3. Estimate parameters of W by recording the correspondence between the crossing points of the original grid image and that of the observed grid image. The parameters express the warping function W. 4. Calculate an inverse warping function 1 W. 5. Prepare an original image I which should be observed linearly by the driver. 6. Make an image of W 1 ( I ) that is going to be projected. 7. When the image W 1 ( I ) is projected on WHUD, the image is distorted at the windshield by the warping function W. Then the distorted image is warped as 1 W ( W ( I)) = I. As a result, driver can see the original image I on the WHUD. Figure 3 shows images of each stage of distortion correction when a grid image is to be projected. Figure 3 Distortion correction process. Transforming an original image I (up left) by 1 W enables drivers to observe the original image I from driver s position. Direction control Another important setup is how to show an image at an arbitrary position. As the screen can be recognized as a simple planner image from the eye point of drivers by the image distortion 4

5 correction, it is relatively easy to calculate the location x given the orientation to be shown to the driver. Suppose the width of the screen is w im and the orientation angle is O when it is parallel to the vehicle s center axis. θ cam indicates a field of view of the screen against the eye position. Then, given the orientation θ, x can be calculated by x = w im tanθ θ cam tan is angle of view of the camera examined in advance and camera image. w im is horizontal resolution of the Figure 4 Orientation from the eye point of a driver VISUAL NAVIGATION ON WHUD System overview As an application that uses the WHUD system, we introduce a visual navigation system. Drivers can see two types of visual information, navigation flags and direction signs, on the WHUD. Navigation flags show the name of a destination and the distance to it. The position of the navigation flag directly indicates the orientation to the destination, so it is easy for drivers to recognize the direction to go. The position of visual information moves appropriately on WHUD as the vehicle moves and makes turn because the orientation is calculated on line. The direction signs show the direction of north, south, east, and west. 5

6 Figure 5 shows two snapshots in driving scenes with the visual navigation on the WHUD. The driver can see the displayed information clearly even in daytime. The navigation flags have to be carefully placed on WHUD so that the navigation flags do not hide an important portion of driver s view of the road scene. In order not to disturb the driver, navigation flags will not be placed on the road area in front of the driver s vehicle. They will be shown at upper or lower region of the WHUD. Figure 4 Navigation flags and direction signs superimposed in a road scene show the orientation and the distance to a destination. The bottom three images are the navigation flag and the direction signs. System components Our visual navigation system requires the WHUD, GPS, and a geomagnetic sensor. The location of the vehicle is estimated by the GPS, and the direction of the vehicle is estimated by the geomagnetic sensor. When the longitude and the latitude of a destination is set, the system calculates the orientation θ and the direct distance L for the destination. 6

7 Δx = ( λ λ Δy = ( φ φ θ = tan d d 1 ) R cosφ ) R Δx θ Δy compass L = Δx + Δy where θ compass is the direction of the vehicle obtained by the geomagnetic sensor, λ and φ are the longitude and latitude obtained by GPS sensor, λ d and φ d are that of destination, and R is the radius of the earth. The formula above is a simple model but it can be used if the destination is not far from the position of the vehicle. System performance In this system, the performance is important because if the frame rate is slow, navigation flags on the WHUD does not move smoothly. This causes a problem especially when the vehicle is making turns. Our system can update the images at the video frame rate of the graphics card of the PC that feeds images to the projector. The PC has Intel Pentium D 3.GHz processor and ATI RADEON X300SE. This means the system can achieve smooth moving of the navigation flags. FUTURE WORK Some experiments to evaluate the system effectiveness and usability are to be conducted. Another work will be the development of applications that uses the WHUD. As we have proposed mixed reality navigation like Virtual Mirror [3] and Virtual Slope [4], they will be the candidates to be realized on this system. Systems that emphasize hazardous objects can also be implemented if a driver viewpoint camera is installed and a proper image processing is done. In the navigation system, the accuracy of sensors, especially that of the orientation estimation, is important. It directly affects to the estimation of the orientation of the navigation flags. With a high accuracy GPS, RTK-GPS, the direction of the vehicle can be calculated by differentiating the position. By the combination of RTK-GPS and geomagnetic sensor, we think the accuracy will be improved. As for the color of projected image on WHUD, green color can be observed well but the other colors are faint. It is because of the material of the windshield that cuts certain range of light wave. However, if we correct the color, many colors can be used for display. 7

8 Figure 5 Experiment to show colors on WHUD. Colors except for green are faint because of the material of the windshield. CONCLUSION In this paper, we introduced a new visual navigation system that shows visual signs on a windshield head-up display, WHUD. The WHUD we have developed has a wide display area and high luminance. Hence it can be used even in daytime. The distortion correction can cancel the distortion effect caused by roundish windshield. The navigation system we proposed shows visual signs on WHUD. The driver can easily recognize the information about the direction and the distance to a destination by the navigation flags. The direction signs helps drivers recognize the direction they go. REFERENCES [1] N. Aoki (004). Perspective for Head-Up Display, Proceedings of 11th World Congress on Intelligent Transportation Systems, Nagoya, Japan, TP , 8 pages. [] K. Nakamura, J. Inada, M. Kakizaki, T. Fujikawa, S. Kasiwada, H. Ando, N. Kawahara (004). Windshield Display for Intelligent Transport System, Proceedings of the 11th World Congress on Intelligent Transportation Systems, Nagoya, Japan, TP , 8 pages. [3] K. Kojima, A. Sato, Y. Kameda, Y. Ohta (005). NaviView: Visual Assistance by Virtual Mirrors at Blind Intersection, Proceedings of 8th IEEE International Conference on Intelligent Transportation Systems, Vienna, Austria, pp [4] F. Taya, Y. Kameda, Y. Ohta (005). NaviVIew: Virtual Slope Visualization of Blind Area at Intersection, Proceedings of 1th World Congress on Intelligent Transport Systems, San Francisco, US, 8 pages. [5] Z. Hu, K. Uchimura, F. Lamosa (004). Towards A New Generation of Car Navigation System-Data Fusion Technology in Solving Camera Registration Problem, Proceedings of the 11th World Congress on Intelligent Transportation Systems, Nagoya, Japan, 1pages. 8