International Symposium on Life-World Semantics and Digital City Design December 9, 2004, Kyoto Research Park SmartWeb: Towards Semantic Web Services for Ambient Intelligence Wolfgang Wahlster German Research Center for Artificial Intelligence, DFKI GmbH Stuhlsatzenhausweg 3, Building 43.8 66123 Saarbruecken, Germany phone: (+49 681) 302-5252/4162 fax: (+49 681) 302-5383/5341 e-mail: wahlster@dfki.de WWW:http://www.dfki.de/~wahlster What is Ambient Intelligence? Ambient Intelligence = Embedding Artificial Intelligence in Everyday Objects and Environments Key Characteristics of Ambient Intelligence: Embedded Situated Personalized Adaptive Pro-active Many networked devices are integrated into the environment These devices can recognize situational context They can be tailored towards your needs and affects They can change in response to you and your task They try to anticipate your plans and intentions 1
Always On and Always Best Connected: Ambient Network Access adhoc networks Sensor & actuator networks Satellites PANs Dynamically changing combinations of network types Wide area networks (cellular) Moving networks Wireless Local Area Networks (hotspot) Home network Fixed access network Wireless extension of coverage Roadmaps for Communication Technologies CELLULAR goes broadband and packet Always best connected (ABC) B3G/4G MLAN (WiMax) INTERNET goes mobile and realtime UMTS WLAN (WiFi) GSM Fixed line Internet 3GPP/3GPP2 IEEE/IETF 2
Ambient Intelligence as an Interdisciplinary Field Nanoelectronics, Mechatronics, New Materials Smart Sensors and Actuators Communication Technology Mobile Networks Computer Science Language Technology Computational Linguistics Artificial Intelligence Cognitive Science Ambient Intelligence is an Integral Part of the Research Mission Living in an Networked World Large Projects on Human-Centered Computing in Germany 1992 SME VR/AR Initiative 20 9 2 Net. of Exc. Morpha Service Robots Basic Research Patents: 11 Embassi Home Electronics Verbmobil Spin Off Products: 20 Smart- Kom Multimodal Interaction Spin Off Enterprises: 8 Invite Distributed Teamwork Map Mobile Interaction Arvika VR/AR Work Environment 9 12 4 6 0 0 52 29 6 1 4 2 12 5 0 37 6 1 Lead Project Service Robotics Lead Project Smart Web 2004-2008 Lead Projects Virtual Human German Future Prize 2001 Artesas 1999 2000 2003 2005 2008 Data provided by Dr. Reuse, BMBF 3
Outline 1. The Architecture of SmartWeb 2. SmartWeb: Semantic Web Services in and between Cars 3. SmartWeb: A Mobile Assistant to Soccer Fans 4. Ambient Intelligence in Future Stores 5. Conclusions SmartWeb-Konsortium The Consortium IMS Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart European Media Lab Funded by the German Future Program : 13.7 Mio Euro Scientific Director: Wolfgang Wahlster Project Duration: 2004-2008 Ludwig-Maximilians- Universität München 4
SmartWeb: The Follow-up Project of SmartKom Mobile Broadband Internet UMTS/HotSpots Mobile Multimodal Dialogue: Speech, Haptics, Gestures New: Open Domain Semantic Web New: Question Answering Intranet with Automated Semantic Markup Webpages With Semantic Markup Language Technology, Information Extraction Traditional HTML- Webpages The Basic Architecture of SmartWeb Mobile Infrastructure Mobile Broadband Services Mobile Multimodal Dialogue using 3G SmartPhones Ontological Infrastructure Top-level Ontology Adhoc Networking between Motor Bike and Car (BMW) Semantic Representation of User Query Information Structures and Media Objects with Semantic Markup Domainspecific Ontologies Peer-to-Peer Services Ontological Decription of Web Services Roaming based on ABC Technologies Access to Semantic Web Services Language-Oriented Ontologies 5
SmartWeb: Semantic Web Service Composition Semantic Web Services OWL-S Web Services UDDI, WSDL, SOAP Semantic Web RDF, OWL WWW HTTP, URL, HTML, XML A Web of Meaning has more Personalization Potential than a Web of Links Three Layers of Webpage Annotations Personalization Potential Content OWL high Structure XML medium Layout HTML low cf.: Dieter Fensel, James Hendler, Henry Liebermann, Wolfgang Wahlster (eds.) Spinning the Semantic Web, MIT Press, November 2002 6
March 2003 ISBN 0-262-06232-1 8 x 9, 392 pp., 98 illus. $40.00/ 26.95 (CLOTH) Edited by Dieter Fensel, James A. Hendler, Henry Lieberman and Wolfgang Wahlster Foreword by Tim Berners-Lee MIT Press Japanese Edition! The Need for Mobile Multimodal Dialogue Systems Broadband mobile Internet access technologies via UMTS or mobile hotspots pave the way for a wide spectrum of added-value web services. but: the user must input more and more complex commands to specify his information needs. PDAs and smartphones with tiny keyboards and mice are useless for mobile settings. Multimodal Dialogue Systems for Mobile Systems 7
SmartWeb Combines Five Input Modalities Speech over Bluetooth Headset Face Camera Gestures based on Pen Input Bio-Signals Haptic Feedback for Handlebars on BMW bike Automated Services in the Semantic Web Automatic Retrieval and Selection of Semantic Web Services Automatic Composition of Semantic Web Services OWL-S Automatic Execution Monitoring of Semantic Web Services Automatic Invocation of Semantic Web Services 8
Spoken Dialogues with the Car Navigation System SmartWeb: Spoken Dialogs for Comparison Shopping in the Car Where do I find within 5 miles the gas station with the best gas prices? I'll guide you to the next gas station with the lowest prices. 9
Mobile SmartWeb Applications for Car Drivers, Motor Bikers, and Pedestrians UMTS Smart Web UMTS DVB-H (Digital Video Broadcasting for Handhelds) W-LAN P2P BMW Bike SmartWeb- Client SmartWeb- Client SmartWeb- Client T-Mobile MDA III Mercedes Car Semantic Peer-to-Peer Communication between Vehicles Safety Agents GPS: Position of Car Attention! Risk of aquaplaning 100 meters ahead!!! GPS: Position of Bike Semantic Peer-to-Peer Connection Aquaplaning Sensor 10
SmartWeb: Multimodal Interfaces to Advanced Motor Bike Services - Radio / CD / DVD Control - Trip Computer - Integrated Chassis Control - Navigation System - SmartWeb Services - Alarms and Warnings UMTS GSM/GPRS WLAN Ad Hoc Network SmartWeb as Mobile Assistant for Soccer Fans at the FIFA Worldcup 2006 in Germany Navigation to the Soccer Stadium Traffic Information around the Stadium Video Streaming of parallel games Question Answering about Soccer Teams and Soccer History 11
Partial View of the Soccer Ontology for the SmartWeb Demonstrator at the FIFA Soccer Worldcup 2006 in Germany Goalkeeper Football player Defender Sweeper Stopper Central defender Substitute Receiver Cap Trainer Ball action Sport-Event-Ontologie Scorer Throw in Clear Kick Pass Volley Score part-of Bar Post Net Ball carrier Goal Referee possesses Ball VIP Football object Ambient Intelligence in the Future Store mixed reality shopping, automatic comparison shopping, cross- and up-selling Fields: smart labels, tracking and tracing, user modelling, plan recognition, location-based services, privacy, security 12
Plan-based Shopping Assistance Instrumented Shopping Cart Instrumented Product Shelf RFID Antenna in Shopping Cart Sample Interaction With Initial Prototype of Integrated System Instrumented shop with tagged items and RFID sensors integrated into shelf and basket User picks item from shelf System retrieves related offers and recommends alternative product User puts item back and takes other product User puts item into basket Interactions can be viewed within Personal Journal 27 13
Example: From Signal to Entry Data Flow Journal Content Sensor Abstraction Methods Personal Journal Signal Level Entry Level Interface <signal name= roomtemperature value= 20 /> <entry entryid= 785121 > <ratings> <affect source= affect?roomtemperature=20 > positive </affect> </ratings> </entry> Journal Entry Schema (Excerpt) 14
Viewer Example: Annotating an Entry References (journal entries, hyperlinks) Comment (free text) Categories (rough content description) Ratings Source Natural Language Conversation with Products in a Cybershop Embedded Speech Recognition and Speech Synthesis Tracking of the User s Shopping Actions 15
Automatic Comparison Shopping with Ambient Intelligence Multimodal Fusion of Speech, Gesture and Physical Actions (Intra- and Extra-Gestures) Symmetric Multimodality for Ambient Intelligence (Wahlster 2003) Multimodal User Input with Emotion Recognition Multimodal System Output with Emotional Coloring Is the resolution okay at least? Now, what do you want to know next? I should be glad to show you special offers. Speech Biosensors Physical Action Speaking Object Presentation Agent Ambient Intelligence 16
Infrastructure 7 Capturing the Cognitive and Affective State of Users by Mobile Biosensors Varioport JAVArio Mobile Biosensors ECG (Electrocardiogram) EMG (Elektromyogram) Muscle Contraction -> Surprise EOG (Electrooculogram) Eyebrows EDA (Electrodermal Activity) Accelorometer Sensor Variable Dynamic Bayesian Network Preprocessing Sensor Data 17
A Four-Layered Architecture for Ambient Intelligence Knowledge Layer Emotional Models Ubi s World Ontologies on Objects, Persons and Activities Presentation Strategies Geographical Representations Service Layer Multi-user Support Shopping Assistance Information Presentation Navigation Services Affective Layer Distributed Multimodal Affect Recognition Hybrid Classifiers Anthropomorphic User Interface Talking Objects Life-like Character Physical Layer Biosensors Shopping Cart PDA Smart Object Instrumented Space Tailoring Mobile Presentations to the Age of the User by Unintrusive User Modeling aging of the human vocal cord tissue Jitter and Shimmer Recognition of the biological age of the speaker s voice using acoustic features Jitter and Shimmer Jesuit Church Jesuit Church 18
COMPASS 2008: Digital Olympics in Beijing 2008 in Cooperation with MOST Comment je fais pour aller du stade à la Cité Interdite? Where ist next bus stop for going to the Great Wall? http://smartkom.dfki.de/ Springer Book about SmartKom: W. Wahlster (ed.) Cognitive Technologies Series 2004 URL of this Talk: www.dfki.de/~wahlster/publications/kyoto_2004/ Smart_Web_Towards_Semantic_Web_Services_For_Ambient_Intelligence.ppt 19
Conclusion The goal of the SmartWeb project is to lay the foundations for multimodal user interfaces to distributed and composable semantic Web services on mobile devices. SmartWeb exploits the machine-understandable content of semantic Web pages for intelligent question-answering as a next step beyond today's search engines. SmartWeb provides a context-aware user interface, so that it can support the user in different roles, e.g. as a car driver, a motor biker, a pedestrian or a sports spectator. 20