Coupled open navigation and augmented reality systems for skippers

Transcription

1 Coupled open navigation and augmented reality systems for skippers J.C. Morgere, Lab-STICC, UBS, France, R. Douguet, Lab-STICC, UBS, France, J.P. Diguet, Lab-STICC, CNRS, France, J. Laurent, Lab-STICC, UBS, France, This paper describes a new measurement and vision system for sailboat race. This system is composed of an open navigation processor and a see-through Augmented Reality (AR) glasses. This open navigation processor allows to plug most sensors in order to measure wind conditions, boat speed, dynamic motions and other parameters. Moreover, it is able to implement several wind correction algorithms in order to improve the true wind computation. On the other hand, the open navigation processor communicates with a see-through AR glasses through wireless network. The interest of this system is that it is flexible and can overlay any text, 2D or 3D objects on the true real world view, so the skipper is free to display the useful data. 1 INTRODUCTION In ocean racing, several domains such as materials, sails and yacht design are constantly evolved in order to improve the boat performance. In order to study the impacts of the proposed improvements, we need to acquire data from the system by using different kinds of sensors (strain gauges, speedometer,anemometer and so on). To achieve these measurements, most sailboats embed navigation processor that allows us to plug and to collect all the data from the sensors. Despite the use of navigation processor, some sensors cannot be plugged since they uses proprietary protocol or yet the navigation processor is too closed to allow plugging sensors that are not provided by the same firm. For instance is difficult to plug NKE sensor with B&G navigation processor. So in order to solve this problem we propose an open navigation processor that implements analogue to digital converters and several communication protocols as, RS232/485, CAN, Ethernet and so on in order to be able to plug most sensors (digital or/and analogue). Furthermore this platform allows the user to implement his own applications as for instance correction algorithms allowing to compute the true wind speed and angle. Another problem is the display of the information collected by the navigation processor to the crew because the displays do not provide the necessary data to analyze the boat performance. Indeed, due to the poor ergonomics and the few viewable data it is very difficult to analyze the boat performance without an embedded PC. The second problem is due to the fact that the crew needs to be able to view the information at different places of the boat. Today, the solution is to deploy several display screens on the boat but this leads cost increase. Furthermore, the weather conditions, as the solar glare or water, can affect the display reading so even the viewable data, provided by the navigation processor, could be unreadable. The solutions considered as the use of tablet is still not the best one because the tablet is not very convenient on boat and reduces the crew mobility since you must use one of your hands to hold it. So, we propose to solve these two problems by using mobile display system called wearable augmented reality system based on a see through device that allows the user to view the real world and also view objects superimposed. This kind of systems can display texts, 2D and 3D objects animated or not. The global solution is presented in Figure 1. Figure 1: System overview The first one presents the open navigation processor, the second the hardware display system and finally the last one an example of the graphical user interface (GUI). The rest of paper is organized as follows: Section 2 describes the open navigation processor. Section 3 and 4 presents the mobile augmented reality system and an application. Finally the Section 5 concludes the paper and presents the future works. 2 OPEN NAVIGATION PROCESSOR The open navigation processor is detailed in three parts: the hardware design (processor, peripherals, sensors), the soft-

2 ware design (Operating System, scheduling tasks) and the wind correction algorithm. 2.1 HARDWARE DESIGN The hardware is the core of the navigation processor. On the one hand, it must be able to receive, to store and to send all data provided by analog or digital sensors. On the other hand, it needs important computing capacities to implement complex algorithms in order to correct different measurements. Indeed, these algorithms require a lot of data processing to compute Kalman-like filters or other kinds of filters. Considering applications requirements a 10 Hz sampling rate is enough to process wind measurements. It also represents an opportunity to save power at the embedded system level. Actually, considering that during a race the fuel required to recharge batteries must be optimized, it is important to reduce the energy consumption. To meet these needs, we chose the STM32F407 microcontroller, which is based on ARM Cortex M4 Processor. This processor includes a floating point unit (to compute complex algorithms) and integrates several communication controllers (UART, SPI, ethernet, ADC...) and general purpose input output in order to plug analog or digital sensors. The hardware design diagram is shown in Figure 2. This hardware platform is also designed for other applications such as, for instance, a databuoy [1]. On the other hand, several peripherals are available to get real time access to data. Indeed, three serial bus are implemented to communicate with other devices. Two serial bus are directly used to send real time data to a computer and to standard displays. But as the STM32f4 microcontroller has not on-board wireless communication, one solution is to add a standard module to interface UART peripheral to wireless. In our case we chose the bluetooth module since it is a wireless networking widely used in smartphone environment for instance smartphone, tablet or computer. Moreover there are Bluetooth Low Energy (BLE) wireless modules which were designed for lowest possible power consumption. The Bluetooth module allows to communique between the navigation processor and the see through AR glasses. 2.2 SOFTWARE DESIGN The navigation processor software design is developed and is implemented through Keil uvision4 Integrated Development Environment (IDE). In order to facilitate a possible extension of this project we chose to divide this application in several tasks (or threads) by using a small footprint Operation System (OS). Indeed the navigation processor software is divided into 4 main tasks : acquisition task, computation task, storage task and communication task (cf. Figure 3). Other tasks runs in background to receive data through interrupts (cf. Figure 4). The selected OS is the FreeRTOS (Real Time Operating System). FreeRTOS is a real-time kernel which is designed for small embedded system. The interests of this OS are the size of kernel, which is limited to 5Kb in flash, it is portable on different supports and it is independent of Keil uvision4 IDE. It allows Cortex-M4 microcontroller applications to be organized as a collection of independent threads of execution by Figure 2: Hardware Design Diagram On this system, we are able to directly plug the standard sensors used on sailboats such as speedometer, anemometervan, Inertial Measurement Unit (IMU) and Global Positioning System (GPS). But many sensors, analog or digital, can be add to this platform since the microcontroller has still several available communication controllers. Indeed this microcontroller proposed up 15 communication interfaces : 3 I 2 C interfaces, 4 USARTs, 2 UARTs, 3 SPIs, 2 CAN interfaces and 1 SDIO interface. Moreover, this open navigation processor allows to record all data at 10 Hz on SD card memory through the SPI bus. Therefore, these data are available for post-processing in order that users can analyze boat performance. Figure 3: Flow chart of main tasks

3 and these measurements are disrupted by several phenomena, such as the boat motions, the drift, the upwash effect and the wind shear [2]. That s why, we have developed our own wind correction algorithm (cf. figure 5) that we implemented on the open navigation processor. So one advantage of this platform is to test differnet wind correction algorithms in order to improve the true wind computation. Figure 4: Flow chart of handler tasks using different tools such as mutexes, semaphores or queues to synchronize these threads Acquisition task This task converts the analog inputs and updates all data provided by the different peripherals. It is executed periodically with a frequency to be specified in the configuration file. This frequency is set to 10 Hz for navigation processor Computation task The computation task allows to calibrate, compute and filter all received data. Currently, it corrects and filters the wind measurement (cf. Section 2.6 for details). 2.3 Storage task This task allows to store all selected data in the file on the SD memory card. These data are logged in the CSV file in order to facilitate their analysis of Matlab, Excel or other softwares. 2.4 Communication task In the communication task, data are sent to several devices through three peripherals. The information sent via UARTs allow to communicate with a computer and with standard displays. Moreover, another UART coupled with a bluetooth module allows to send data to different devices such as smartphones, tablets and laptop and more specifically in our case to the see-through AR glasses. 2.5 Handler Tasks The handler tasks are synchronized by a semaphore with the interrupt functions. They allow to receive the data provided by sensors with serial communication. For example, when the processor receive a GPS packet, a semaphore is given in the GPS interrupt function and then the GPS task is unblock in order to parse the GPS packet. 2.6 WIND COMPUTATION Usually standard wind corrections are handled by navigation processors on sailboats where true wind is required. Indeed, the wind sensor is placed on the masthead most of the time Figure 5: Wind Correction Flow 3 MOBILE AUGMENTED REALITY 3.1 HARDWARE DESIGN The augmented reality device is composed of two parts, the first one is the display and the other is the embedded processor. The display is a see-through ski mask [3] which includes an electronic and optic system that diffuse frames in front of an eye (monocular system). In order to facilitate the reading information, the display system has the following features: The resolution of the mask is 800*600 pixels with a refresh rate at 60Hz The equivalent screen size is 97.5 inches at 2.7 meters so a large amount of data can be displayed and readable The light intensity is 5000 candela/m 2 in color mode (greater in monochrome mode) so the information displayed will be able to be visible even in outdoor.

4 The second part of the system is the embedded processor. For the mobile augmented reality, this processor can be ARspecific embedded and reconfigurable systems [4]. This is a low consumption and small size architecture (due to the number of components needed) and it is dedicated to augmented reality systems with head tracking and using very small definition objects. Head tracking is employed to place objects in the user field of view, for this step MEMs are often used with correction algorithm to compute the system attitude. For better performance and consumption it would be necessary to produce and use Application-Specific Integrated Circuit (ASIC) but this technology is very expensive and not interesting for few units. But today this kind of solution is not usable since we need to offer to the user a large amount of objects that can be simple key arrow or text but also more complex objects like 3D texturing sailboats. So the AR-specific architecture performances are not sufficient this is the reason why we chose a classical SoC solution used for instance in commercial tablets or smartphones; the system architecture is shown in Figure 6. In our case, we use MEMs sensors: one magnetometer, one accelerometer and one gyroscope for the head tracking. Despite the use of these sensors, we must, in addition, execute a correction algorithm (usually an extended Kalman filter) in order to obtain a precise attitude. Some information, as GPS localization, are also required so in most case we will have to discuss with the navigation processor to obtain this data; this will be done by using a wireless communication (WiFi, Bluetooth). For low energy consumption criteria, we chose to use Bluetooth communications. To store the data, the applications and the OS we need memory capacities thus in our system we have emmc, DDR2 and SD-Card; emmc and SD-card are non volatile memories so applications and OS will be able to be implemented into one of these memories. Although the SD card is bigger than the emmc, this last would be used in order to increase the execution time, indeed the emmc is faster than the SD-card. Finally, as the embedded processor has to be connected to the ski mask, it s necessary to use a video connection in order to display the virtual objects on the mask. Today the ski mask uses an analog video stream (VGA) so as the SoC system generates a digital one (HDMI), we must convert the HDMI stream with a converter (will be change in future works). 3.2 SOFTWARE DESIGN Figure 6: Architecture SoC A9500 ST-Ericsson for Mobile Augmented Reality device This architecture is a System on Chip (SoC) designed for mobile applications running on embedded operating system (EOS) and composed of: General Purpose Processor (GPP) that executes the operating system (android in our case), Graphical Processor Unit (GPU) that allows us to generate and manipulate the graphical objects, Image Specific Processor (ISP) that manages the video stream (does not us in our case), Video that performs some decode step for compression/decompression algorithm, Peripherals that allows us to connect our system with others The GPP is composed of two ARM Cortex A9 whose the internal frequency can vary from 200 MHz up to 1.0GHz so it s very interesting for power/energy management. The GPU [5] can process about 1.1 Gpixels/s so these performances are sufficient for our target applications. However, to be able to place the virtual objects for the augmented reality system, we need sensors allowing to acquire 9 degrees of freedom (DoF). The software part of the system requires to manipulate different levels of abstraction from the physical level (sensors) to application level (3D objects for instance) so in order to facilitate the programming scheme, we implemented an EOS (Embedded Operating System) called Android. This EOS, thanks to the many APIs available like OpenGL ES 2.0 for GPU instructions, sensor manager for MEMs, location manager for GPS, and WiFi manager and Bluetooth adapter for wireless connectivity, allows us to rapidly develop our augmented reality application. Furthermore we can develop the application by using JAVA and/or C/C++ (for native applications) so in the same application, some parts can be realized in JAVA and communicate with others developed in C/C++. The software architecture is composed of four parts, a time service to read and control number of frames per second of the application, graphic service to send commands to the GPU, an input service to read sensor and wireless events and the last one is orientation service to compute position and orientation objects. Graphics service must read and load texture and shader objects, configure OpenGL ES parameters and send a draw command to the GPU. Input service is used to read and write data to the Bluetooth connection in a dedicated task whereas another task (looper task) is used to read sensor events and saves data. If head tracking added for geographic positioning, input service could add a special task for orientation computing with a filter (Gradient descent, Kalman, Extended Kalman filter and so on) to correct MEMs data and compute attitude. The orientation service updates orientations and positions of all objects, it can delete or add new objects (depends of the application) and can compute AIS data ( boat position, heading, and name) to collision detection.

5 TWS kt 12.3 BSP 10.0 TWA deg 43.0 kt HDG deg 135 Figure 7: Prototype 4 APPLICATION application can display the boat polars so that everyone on the boat can have the boat performance to check and validate The aim of our application is to help crews during the ocean the sail choice and adjustment. races or inshore. So the first application is dedicated to the skipper thus we display only text which gives him three kinds Finally the last application (Figure 8 example D) can be dedof data. The first one concerns the boat and gives the heading, icated to the tactician since during inshore it can be precious speed, heel and drift. The second gives information about the to be able to see the buoys and the positions of the other boats wind; for both the apparent and true wind, we gives the speed in order to take decisions. All the necessary data can come and angle. Finally, we display various data come from GPS from the navigation processor (GPS and AIS data) via a wireas the position, the time and so on. Each data are differentia- less communication as Bluetooth for instance. With our open ted by a different color in order to respect the readability and navigation processor this application is very easy to develop. ergonomic; this solution is shown in Figure 8, example A. 5 CONCLUSIONS In this paper, we present a new open solution to measure and visualize the sailboat performances. Our system has the advantage of flexibility since the navigation processor allows to plug most sensors and to easily implement new computations (by adding new tasks). On the other hand, the glasses display is totally flexible and allows to visualize all data or map navigation according to the skipper choices. Currently we have a prototype as illustrated in Figure 7. A box (Pelicase) contains the embedded system for the navigation processor and standard sensors such as anemometervan, speedometer, GPS and IMU can be directly plug it. All data are sent to the see-through AR glasses through a bluetooth link and we can visualize them in real-time. The next step of the project is to improve the see-through AR glasses by integrating all the embedded system (electronic The second application allows us to provide for each crew board and battery) in this display device. This is, we study members information in function of his job on the boat for ins- the energy consumption of this board to optimize power contance for the wind presentation, we display a disc with sailing sumption in order to reduce the battery size and weight. boat textured and arrow keys moving around it according to the wind angle; for the wind speed we use text form (Figure 8 REFERENCES example B). Another way to display the information can be the use of 3D boat in order to represent the heel, the trim and [1] R. Douguet, J.P. Diguet, J. Laurent, Y. Riou Open Data the heading and the use of key arrows for showing the appabuoy to Analyze Weather and Sea Conditions for Sailing rent and true wind angle (Figure 8 example C). The second Regattas, OCEANS 13 MTS/IEEE, Bergen, NO, Figure 8: Examples of displays

6 [2] R. Douguet, J. P. Diguet, J. Laurent, Y. Riou A New Real-time Method for Sailboat Performance estimation based on Leeway Modeling, The 21 Chesapeake Sailing Yacht Symposium, Annapolis, MD, March [3] Laster Technologies, [4] J. P. Diguet, N. Bergmann, J. C Morgere, Embedded System Architecture for Mobile Augmented Reality (sailor case study), PECCS13, Barcelone, ES, [5] ARM, Mali-400 MP, arm.com/products/multimedia/ mali-graphics-hardware/mali-400-mp. php [6] A. Pons, D. Asiain, F. Quero, J. Cuevas, Racing Bravo. Un sistema de navigacion para alta competicion, Madrid Diseno de Yates 2004, Madrid, ES, AUTHORS BIOGRAPHY Jean-Christophe Morgère is a PhD student at Lab-STICC, Université de Bretagne Sud, he obtained his Master degree in Electronics. He is now working on design of a mobile augmented reality embedded system: multiple sensors, low consumption and prototype in marine domain. Ronan Douguet is a PhD student working between the Groupama Sailing Team and the Lab-STICC laboratory in Lorient, he obtained his Master degree in Electronics. His work concerns the sailboat performance analysis and more specifically the improvements in wind measurements. Jean-Philippe Diguet is a CNRS Research Director at Lab-STICC. He has multiple research activities which referred to modelling and electronic development in embedded platforms. His previous experience includes several thesis supervising in embedded systems and some marine specific topics about sailing performance or on-board mobile augmented reality. Johann Laurent works as an Associate Professor at Lab- STICC, Universite de Bretagne Sud. He is in charge of teaching in a Master degree in electronics and have research activities which deals mainly with the power energy consumption in embedded systems. He supervises several works around electronic research for sailing including thesis and internships with manufacturers and sailing teams.