A Novel Interactivity Environment for Integrated Intelligent Transportation and Telematic Systems

Transcription

1 Submitted to The 5th International IEEE Conference on Intelligent Transportation Systems, Singapore, September 3-6, 2002 A Novel Interactivity Environment for Integrated Intelligent Transportation and Telematic Systems T. Brett Hall and Mohan M. Trivedi Computer Vision and Robotics Research Laboratory University of California, San Diego La Jolla, CA Abstract Advances in intelligent transportation are rapidly sensorizing and connecting both highways and vehicles. This recent availability of high-speed communication and large quantities of information are rapidly changing the way we perceive transportation. In order to develop technologies and applications that are practical in this intelligent environment, it is important to consider specific ways to best use this infrastructure effectively. This paper focuses on the development of and experiments with a visualization environment that presents an interactive, immersive, and context-preserving display of outdoor information. The developed visualization integrates a variety of types of information about a section of interstate such as aerial photographs, maps, and live outdoor videos into a single interactive environment. Additionally, a wireless mobile testbed is presented that allows the undertaking of novel interactivity experiments among mobile participants without requiring a fully instrumented vehicle. Together these systems allow effective experimentation with and integration between highway and vehicle based real-world intelligent transportation systems. Index Terms Telematics. Traffic Monitoring, Human- Computer Interactions, Computer vision, Televiewing, Graphical Interfaces I. INTRODUCTION AND MOTIVATION Intelligent transportation systems include both infrastructure and vehicles. In order to take full advantage of this new world of intelligent vehicles operating on the intelligent highways, new tools and techniques are required to allow for most effective means of interactivity. Potential users of these new technologies will be numerous. For example, they could be police or an emergency response team member or they could be individuals (from a stationary site or from another vehicle) who are interested in having a conversation with the driver, in a safe manner. It is very important to maintain the proper spatio-temporal context in supporting such dynamic interactions. In these interactions, maintaining the remote users' "situational awareness" is important in readily conveying the state of the automobile to other parties across a communication link. This paper addresses such issues and provides some tools for making these interactions possible. We present two main ideas, (1) a context aware map for fusing static maps, aerial images, and live video streams for situational awareness, and (2) a mobile testbed, called MIA for mobile interactive avatar, which allows us to undertake a wide range of experimental trials involving bi-directional video and audio based interactions among mobile participants over high bandwidth wireless links. This paper will include discussions of ongoing experiments involving these systems deployed on the UCSD campus and more recently on the Interstate I-5. A diagram of a highway scenario is depicted in figure 1 indicating the presence of intelligent equipment in the vehicle and on the highway. The requirements of an environment enabling situational awareness are important to consider in communicating a visually appealing and effective representation of the remote space. First, a variety of types and resolutions of content-rich information are needed to adequately represent the space and the appropriate details within the space. Second, the application should be interactive and allow the user to specify which areas, scales, and information they are interested in seeing at that instant. These transitions between requested views should be smooth in order to maintain user orientation. Third and most importantly, the information needs to be presented together with its surrounding environmental context for most effectively conveying this large quantity of information to the user and maintaining an overall sense of togetherness. The MIA also has essential requirements in that it can be instrumented to test mobile dynamic interactions. It needs to be mobile and capable of operation indoors and out. The MIA also should involve technologies under consideration for use inside vehicles including high bandwidth wireless communication, audio devices, and both video sensors and video displays. 1 CVRR lab website: contact: bretthall@ieee.org, mtrivedi@ucsd.edu 2

2 DIVA: Distributed Interactive Video Arrays Figure 1:This environment surrounding this research involves intelligent highways and vehicles including distributed interactive video arrays and wireless coverage along the highway, and a multitude of in-car systems including vision sensors, microphones, GPS, and wireless networks. II. CONTEXT AWARE MAP BASED INTERACTIVITY An architecture has been developed for the organization and development of the environment; this architecture is pictured in figure 2. It is useful to begin to understand the relationships between the functional elements and the information to get a sense for the system organization. The core element is the context aware map that stores and presents a variety of information about the environment interactively to the user. This information includes maps, but is also able to include aerial images, CAD drawings, and live video information. An important assumption of this environment is that the ground is represented as a flat, 2-D surface within the application. This assumption reduces the computational complexity and enables incorporation of live information into an interactive visualization that runs on standard computer platforms. By requiring simply a 2-D georegistered aerial image to begin instead of 3-D data, the processing, registration, and rendering steps are faster while still creating an effective visualization. The map is referred to as context aware because it uses a diverse array of information to present the context for a space. Context is defined as the circumstances in which an event occurs; a setting. For use in this visualization, representation of the environmental context can be achieved through the application of multiple pieces and resolutions of information about an area, each of which has a natural resolution or scale associated with it. By using multiple pieces of information at different scales, the context can be captured and represented. However to present multiple pieces of information together in a comprehensible format a 3 physical ground basis is needed to set the relative information positions and scales. This process of georegistration can be very complex; however using an orthographically registered aerial image as a basis is a good approximation. Information registration with respect to this background image is accomplished using a simple 4-point registration method. III. TELE-EXPLORATION AND TELEVIEWING FRAMEWORK A framework and system organization beyond what the architecture specifies is necessary for system development. The established hardware framework enhances the modularity and flexibility of the resulting system as well as illustrating how the elements are assembled into a working system. A key component of this framework is the division of the elements; primarily the separation and networking of the sensing and processing elements. This decreases the dependence on a central coordination agent and results in a more flexible and robust system for the application contexts being considered. This distribution of key elements to physically separate but networked locations results in the ability to run the user interfaces and computationally complex algorithms from different locations based on available resources and needs. The context aware interactive map can be run in multiple locations for distributed users because it relies only on a network connection to connect to the essential elements. The mobile aspect of this framework incorporates the control and bidirectional information transmission of mobile platforms into the strategy using a combination of wired and wireless networks. The implementation of this framework is objectoriented and modular such that the elements can be re-used

3 Figure 2: The system architecture has distributed sensing and processing elements to support the context aware map based interactivity in different implementations. An important feature of this organization is the distribution of the video sensing elements; the video for the visualization is captured and transmitted over the wired and wireless networks. This removes a conventional constraint by eliminating the need for a video cable connected directly from the camera to the main computer. The requirement of information acquisition is important to satisfy early in the development of the context aware map environment because of the need for infrastructure development. Providing live information to an interactive application requires a careful system organization and a variety of sensors. IV. STATIC AND DYNAMIC STREAMS OF INFORMATION Achieving a good understanding of a remote space requires using appropriate content-rich information as a representation of the space. There are a couple distinct categories of information that are specifically considered for use in the representation of the space: static and dynamic. Static information includes many different and readily available types including perspective and aerial images, road maps, CAD drawings, etc. This kind of information provides both appearance-based background and foreground information as well as symbolic and other non-appearance based information. Dynamic information includes information that changes over time such as live video, weather, wireless network coverage, etc. Primarily live video is being utilized as dynamic information source. See figure 3 for some of the kinds of information that are considered within this framework. 4

4 Figure 4: The Video on Wheels testbed uses multiple modalities including rectilinear, omni-directional, thermal, and tri-nocular stereo cameras to capture the state of the driver, passengers, and interior of the vehicle Figure 3: Aerial images, maps, and live video contain contentrich information processors, and bi-directional communication links to help perform a variety of tasks related to driving. The development of an extendable architecture for fixed indoor/outdoor visual sensor installations was discussed in [2] but will be briefly mentioned here for completeness. To develop and test this environment a research testbed of video clusters has been designed and installed at the UCSD CVRR. Currently several visual sensor clusters exist outside our lab near a busy intersection on campus where buses, cars, bikes, and people are regularly in motion. Our newest cluster has a direct view of I-5 from on campus. Each sensor cluster contains a high-speed pan/tilt/zoom (PTZ) rectilinear camera and Omni-Directional Video Sensor (ODVS) both mounted in weatherproof housings. All of this video is digitized by special video server units and streamed to the lab using direct fiber optics and a pair of one-gigabit switches. This special high-speed network connection has the capability of carrying sixteen full-size video streams simultaneously. This video is also switched to the public internet from the lab, allowing global access to the video cluster information and PTZ camera control. This distributed sensing element enables the visualization to be run with live video anywhere network is available. V. TESTBED FOR MOBILE MEDIA AND TELEMATICS The primary objective of our multiple mobile media and telematic testbed developments is to capture and display multi-modal information from moving vehicles. To further accomplish research in this area, we needed some representative platforms that we could appropriately instrument and test. Two novel testbeds have come out of this effort; pictured first in figure 4 is the Video on Wheels (VoW) testbed. This car frame with multiple vision sensors installed enables vision experiments within the interior of a vehicle to be performed without the complexity or expense of instrumenting a real vehicle. [17] The second testbed is the Mobile Interactive Avatar (MIA), shown in figure 5. Based on the current industry direction, it seems clear that future cars will have increasing numbers of telematics devices installed and integrated as part of the vehicle. The vehicle occupants will have access to these in-car sensors, 5 Figure 5: The MIAs capabilities of wireless bidirectional video and audio transmissions enable telematic interactivity, telepresence, and remote sensor placement experiments to be performed within in sensorized, intelligent environments. The MIA is a testbed with many telematic systems integrated onboard (see figure 5) including rectilinear and omni-directional video-capture, audio capture, a video display, an audio speaker, self-localization, a laptop computer, b wireless networks, and an indoor/outdoor mobile base providing power and mobility. The onboard systems form the basic abilities of the MIA including bi-directional audio and video transfer with the

5 teleguide (operator) and other people interested in the avatar s location. The onboard video display and speakers are for seeing and hearing the person the avatar is representing remotely. The high mounting location of the b antenna helps maintain a high bandwidth wireless link for the system during operation. Additionally the MIA is capable of self-localization using a high resolution differential GPS (Global Positioning System). All of these features make the MIA an essential testbed for doing real experiments without requiring the full-scale automobiles. These experiments may result in novel systems that get integrated in future automobiles. The MIA enables the early development and systematic evaluation of novel mobile user interaction concepts under real-world conditions as indicated in figure 6. The MIA also enhances the context aware map by enabling live and immersive viewpoints of a space. incident detection, traffic flow management, and nearby vehicle environment detection as demonstrated in figure 6. A visualization using the newest sensor cluster along I-5 corridor where it passes through UCSD campus has been developed for these studies in figure 5 and 6. As a component of ITS, this visualization enables viewing of live traffic conditions within context of the surrounding roadways and areas. Multiple new video clusters are being installed along interstates as infrastructure to improve the accuracy and speed of traffic incident management. As part of that activity, this research is being used to provide the ability to monitor and explore large areas of outdoor roadbed. This implementation of the visualization runs on standard PC hardware under Windows using conventional networks for information transmission. The resulting visualization is dynamic and interactive, which is very difficult to summarize and describe in a static document. Some screen shots of the application running are included to attempt to demonstrate some of the capabilities of the application. Figure 6: The MIA offers multimodal bidirectional wireless connectivity on a mobile planform. The MIA is designed within the framework of sensorized environments that reduce the expense and complexity of the onboard equipment. The MIA relies upon wireless network and visual sensor coverage of its current position to simplify the navigation and communication tasks. This sensorized and networked environment also fosters research into interfaces with which to interact and control multiple smart, distributed systems. VI. EXPERIMENTS The development of this visualization framework and application displays a concept of a useful exploration environment that will be available in future generations of portable mapping devices. With the rate at which current computing power, mobile systems, and graphics acceleration are increasing, future mobile devices will have system resources beyond what is currently available on stationary, larger computers. With the future possibilities in mind, the existing implementation is targeted for powerful desktop computing technology to enable the demonstration of real-time and interactive implementations. The environment has been demonstrated to be useful in outdoor visualization for ongoing experiments in traffic 6 Figure 7: This screenshot demonstrates the results; live I-5 video is viewed within the background (which is a static highresolution aerial image) The application gives the user the ability to interactively view the environment easily by smoothly zooming, translating, and rotating the scene providing continuity to the visualization as the view changes. Multiple layers of static and dynamic images convey both visual and symbolic information within the context of the surrounding area. Each individual image/video can be fully manipulated using this interface by clicking and dragging on the right hand side icons. Additionally the images are selectable by clicking directly in the interface window. In addition to the experiments with the visualization, the experiments with the MIA have been very successful. After being built and tested it has demonstrated the ability to act as a vehicle for communication between the teleguide and remote and mobile outdoor people. Roaming in-between indoor and outdoor areas under coverage of wireless

6 High-Resolution Aerial Image Background MAP Live Video Figure 8: A screenshot of the interactive application shows the viewing of maps, high-resolution aerial images, and live video; each within context of the surrounding area. This static capture shows I-5 running through UCSD campus. networks has been accomplished, as well as interactive viewing of remote spaces using onboard sensors. Even early experiments with the MIA have established the need for additional research in wireless, video, audio, and interactivity areas. The MIA has also served as a base for in-car videoconferencing experiments while driving within approximately a ½ mile of I-5 that was been instrumented with b wireless coverage. The high-bandwidth wireless technology successfully enables full color large images to be exchanged over a short range of highway. VII. CONCLUDING REMARKS This powerful environment that has been developed provides a way to methodically acquire, process, and intuitively represent a variety of information types to the user based on their interest. The framework has provided a flexible and robust methodology in the implementation of this environment. The resulting application successfully allows a user to interactively view outdoor spaces and highways using multiple types of static information as well as multiple live video elements. This visualization is particularly effective because of its presentation of the context around the area of interest. Additionally the MIA has proved to be a valuable testbed for telematics system integration and mobile interactivity experiments. This 7 research is currently being extended to several new applications including in-car systems and interfaces, bridges, multi-sensor cluster interstate coverage, and border-crossing. VIII. ACKNOWLEDGEMENTS Our research is supported in part by the California Digital Media Innovation Program (DiMi) in partnership with the California Department of Transportation (Caltrans) and the DaimlerChrysler Research and Development center. We also wish to thank our colleagues from the lab who are also involved in related research activities. We would especially like to thank Greg Kogut, Andrew Cosand, Ofer Achler, and Natalie Roselli for their assistance in the development of our I-5 and VoW testbeds. IX. REFERENCES [1] B. Hall. Visual Tele-Exploration using Multiple Cameras and Mobile Avatars: An Integrated Framework and Experiments Masters Thesis, Electrical Engineering, University of California San Diego, December 2001 [2] B. Hall, K. Huang, M. M. Trivedi. "A Televiewing System for Multiple Simultaneous Customized Perspectives and Resolutions," Proceedings of the Intelligent Transportation Systems Council, Oakland, August 2001.

7 [3] M. M. Trivedi, B. Hall, G. Kogut, S. Roche. Web-based teleautonomy and telepresence, Proceedings of SPIE Vol. 4120, p , San Diego, August [4] R. Capella. " Real-Time System for Integrated Omni-Directional and Rectilinear Image-based Virtual Walkthroughs," Masters Thesis, Electrical Engineering, University of California San Diego, [5] K. Huang, M. M. Trivedi, "Networked omnivision arrays for intelligent environment", Proceedings of the Applications and Science of Soft Computing IV, August 2001 [6] M. M. Trivedi, S. Bhonsle, A. Gupta, "Database Architecture for Autonomous Transportation Agents for On-scene Networked Incident Management (ATON)," International Conference on Pattern Recognition, Barcelona, Spain, September 2000 [7] K. Ng, H. Ishiguro, M. M. Trivedi, T. Sogo. Monitoring Dynamically Changing Environments by Ubiquitous Vision System, IEEE Visual Surveillance Workshop, Fort Collins, Colorado, June [8] S. Hsu, S. Samarasekera, R. Kumar, H.S. Sawhney. Pose Estimation, Model Refinement, and Enhanced Visualization Using Video, Proceedings of CVPR 00, Hilton Head Is., SC, vol. I, pp , June [9] S. Teller, M. Bosse, D. de Couto. Eyes of Argus: Georeferenced Imagery in Urban Environments, GPS World, pp , April [10] L. Lee, R. Romano, G. Stein. Monitoring activities from multiple video streams: establishing a common coordinate frame, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.22, (no.8), IEEE Computer Society, August [11] R. Collins, A. Lipton, T. Kanade, H. Fujiyoshi, D. Duggins, Y. Tsin, D. Tolliver, N. Enomoto, O. Hasegawa. A System for Video Surveillance and Monitoring, tech. report CMU-RI-TR-00-12, Robotics Institute, Carnegie Mellon University, May, [12] R. Kumar, H.S. Sawhney, J. C. Asmuth, A. Pope, S. Hsu. "Registration of video to geo-referenced imagery," Proceedings of ICPR 98, Brisbane. Australia, Aug [13] K. Venkatesh Prasad, "The emerging fabric of seamless mobility," Proc. Society of Automotive Engineers Annual Conf, 200. [14] S. Ashley, "Driving the info Highway," Scientific American, October [15] "Ubiquitous mobile interner focus for Intelligent Transport," Scientific American Newsletters, January [16] M. M.. Trivedi, I. Mikic, G. Kogut, "Distributed Video Networks for Incident Detection and Management", IEEE Conference on Intelligent Transportation Systems, Dearborn, Michigan, October [17] I. Mikic, M.M. Trivedi, Vehicle Occupant Posture Analysis for Smart-airbags ITS World Congress, Chicago, 2002 (Under review) 8