The Need for Mobile Multimodal Dialogue Systems

Transcription

1 The Need for Mobile Multimodal Dialogue Systems Broadband mobile Internet access technologies via UMTS or mobile hotspots pave the way for a wide spectrum of added-value web services. but: the user must input more and more complex commands to specify his information needs. PDAs and smartphones with tiny keyboards and mice are useless for mobile settings. Multimodal Dialogue Systems for Mobile Systems

2 Multimodal and Multimedia Interfaces A multimodal/multimedia interface is is one one that that supports more than than one one mode/medium through which users and and application programs can can communicate. For For example such an an interface might support: -graphics -gestures -natural language -menus -sketching -video -animation -sounds Medium::= physical carrier of of information (eg. (eg. Acoustic versus optical) Mode::= particular sign sign system (eg. (eg. Language versus graphics) Examples: natural language and and graphics output on on a computer display multimodal presentation Speech and anddataglove Input multimedia input

3 Using All Human Senses for Intuitive Interaction: Code, Media and Modalities language graphics gesture facial expression MEDIA CODE (systems of symbols) (physical information carriers) Output Channels Input Channels visual MODALITIES (human senses) auditory System Storage HD Drive DVD tactile User haptic

4 From Multimedia to Intellimedia Conventional Multimedia Systems no no representation of of the the content of of the the presentation predesigned text text and and graphics fixed presentation sequence Hypermedia Systems no no representation of of the the content of of the the presentation simple semantic relations between parts of of the the presentation finite hyperspace of of alternate presentation Intellimedia Systems representation of of the the content of of the the presentation no nopredesigned text text and and graphics infinite number of of customized presentations

5 Limitations of Current Multimedia Technology Although multimedia technology is is so so popular, that that virtually everyone is is using it it in in some form, the the technology is is still still in in its its infancy: one canned presentation segments hardwired presentation sequences very very limited limited user user adaptation inadequate media media coordination no no unanticipated presentation possible no no follow-up questions possible very very limited limited user user interaction no no high-level authoring tools tools no no content-based reuse reuse strategies no no inference services no no representation of of the the content of of the the presentation no no formal formal semantics/pragmatics of of presentations AI AI can help to to solve these fundamental problems of of current multimedia technology

6 AI Meets Multimedia Technology Knowledge Representation Planning Temporal Reasoning MULTI Media Allocation Presentation Design Media Coordination A I Image Understanding Speech Understanding Text Understanding Language Generation Graphics Generation Music Generation User Modelling Discourse Processing Analysis of Multimodal Input Generation of Multimodal Output Adaptive Presentation Multimodal Interaction Constraint Processing MEDIA Presentation Layout

7 Guidelines for an Intelligent Multimedia Interface 1. Careful coordination of different media in a coherent and cooperative dialog Coexistence of Input and Output in Different Media = Effective User Interface for Decision Support 2. No interpretatioin and no presentation without representation Input / Output without Representation

8 The Role of Representation Languages in Intelligent Man-Machine Interfaces Image Synthesis Images Image Understandig Image Description Language Parsing/ Semantic Construction NL Utterance Generation Semantic Representation Language Gesture Analysis Gesture Gesture Generation Gesture Description Language External Data Source KR Language Inference Engine DBMS/ KBMS

9 Generating and Transforming Presentations in Different Modes and Media Text Text Concretion ANIMAL, ANTLIMA, SPRINT LANDSCAN, VITRA, NAOS Graphics // Images VERBALIZATION KAMP PAULINE POPEL TEXPLAN Abstraction 100% 100% APT APEX ESPLANADE Text & Graphics BETTY Multimodal Presentation COMET WIP / PPP SAGE, AIMI Formal Representation of of Information to to be be Conveyed VISUALIZATION

10 Knowledge-Based Design of Multimedia Dialogs Task Pipeline Design of Dialog Structure Design of Presentation Structure for Conveying Communicative Acts Design of Media Objects Knowledge Sources Dialog Dialog Strategies Presentation Strategies Media-Specific Design Design Strategies

11 The Seeheim Model (Pfaff 1985): Classical Approach to User Interface Management Systems Application User Application Interface Dialog Dialog Control Presentation Layer Layer

12 What are Agent-based User Interfaces? appear as life-like characters plan interactive behavior autonomously can initiate interaction ACTIVE understand the user s requests answer clarification questions allow mixed initiative dialogs INTER- ACTIVE INTERFACE AGENTS RE- ACTIVE respond immediately to interruptions criticism and clarification questions direct manipulation PROACTIVE anticipate the user's needs adopt the user's goals provide unsolicited comments

13 The Architecture of the SmartKom Agent (cf. Maybury/Wahlster 1998) Input Processing Media Analysis Media Analysis Interaction Management Language Media Fusion User(s) Graphics Gesture Biometrics Media Design Design Language Graphics Gesture Discourse Modeling Intention Recognition User Modeling Application Interface Information Applications People Output Rendering Animated Presentation Agent Media Fission User Model Discourse Model Domain Model Task Model Representation and Inference Media Models

14 Mark Maybury, Wolfgang Wahlster (eds.): Readings in Intelligent User Interfaces Morgan Kaufmann Publishers, April 1998, 672 pages, ISBN

15 Multimodal Interfaces to 3G Mobile Services Market studies predict: Cumulative revenues of almost 1 trillion from launch until average 3G subscriber will spend about 30 per month on 3G data services Multimodal UMTS Systems 322 billion in revenues in 2010 Non-voice service revenues will dominate voice revenues by year 3 and comprise 66% of 3G service revenues by 2010

16 Mobile Messaging Services Evolution: From SMS to MMS Customer Expectation Location-based Services Personalized Services Limited Enhancement MMS EMS Multimedia Enhanced Text Video Applications Emotional Experience Youth Focus Ubiquity SMS Text Pictures Audio Integrated Image Capture Enhanced Message Creation Standard Phones SS7 SMSC SS7 UMTS Smart Phones EMS Phones SMSC IP/MPLS Protocols Terminals MMS Phones MMS Relay and Servers Infrastructure

17 The Need for Personalization: Adaptive Interaction with Mobile Devices? e.g * 768 pixel 24-bit color e.g. 60 x 90 pixel b/w

18 Personal Area Networks (PAN) for Mobile Services Bio sensors for medical surveillance 2,50 V 80µA Fibers for Power Electronic Communication MEMS % of consumers saying a capability would be useful Refrigerator capability to track content Appliances that communicate when in need of service Message on TV indicating washing machine is finished Distributed audio (across home) Turn appliances on/off via mobile phone alerts when deliveries are made source: Parks Associates

19 Towards Ambient Mobile Multimodal Systems Ambient Mobile Multimodal Systems Mobility Speed for information access Radio Individuality Widespread usage of the Internet Cost reduction Pervasive communication Multimedia services Network Variety of devices Services Multimodal Interaction Social trends Economical trends Technological trends

20 Vision: RFID-based Payment and Location Tracking

21 Wireless Sensor Networks (WSN) as a Basis for Ambient Intelligence Small form factor, low power wireless microprocessor, RF and sensor components; sensor I/O. Totally self-organizing, self-healing networking software. A network of things. Data infrastructure typically in place; web services based.

22 The WSN Landscape Power & Data Rate Power (mw) WiFi b WiFi a, g LAN 100 Bluetooth BT1 BT2 10 PAN 1 ZigBee k 100k 1M 10M 100M Data Rate (bps)

23 Technology components for Wireless Sensor Networks Endpoints Direct interface to analog and digital sensors and control points Routers Extend range of network Form redundant routes Gateways Transfer data among endpoints and higher-level system Monitor data links and battery status Software Development, Deployment and Management toolsets

24 Monitor for the Ad-hoc Networking of I-Bean Sensors

25 Network topologies for WSN: Star Star uses simple, low-power endpoints.

26 Network topologies for WSN: Mesh Mesh uses routing nodes to extend coverage area with nodeto-node hopping

27 Network topologies for WSN: Star/Mesh Star/ Mesh uses meshed routers to extend coverage and route around obstacles with star endpoints that consume very little power.

28 Features of WSN Technology Small form factor Ultra-low power Battery life measured in years Battery Type Transmit CR2032. CR2477 Rate 220 MA hour 1000 MA hour. 1.0 Hz 28 months 1 year 0.1 Hz 2.1 years 9.5 years 0.01 Hz 4.5 years 10+ years* *battery shelf life Robust, efficient and flexible protocol. Application classes Periodic Sampling Event Driven Store & Forward Topology choices to fit application requirements Star Mesh Star/Mesh Hybrid

29 Targeted Applications of WSN Building Automation Temperature/Humidity Monitoring Occupancy Monitoring Damper/Fan/Valve Control Industrial Automation Pressure/Flow/Temper ature Monitoring Machine Condition Monitoring Metering Gas/Water/Electric Security Rail/Bus Monitoring Traffic Monitoring Infrastructure Monitoring Supply Chain Management Tracking Goods in Transit Remote Health Management

30 Wireless Sensor Network Communications Inside and Outside the Home. Energy Management Gas Water Electric Central Monitoring / Internet Gateway Choices: Set top Box Cable Modem / DSL Cellular Other Wireless Gateway Self-Organizng Sensor Network Security Monitoring Environmental Monitoring Personal Health Monitoring Security Sensors Environmental Sensors Bio- sensors Maintenance Management

31 Monitored Fire Extinguisher based on WSN 6th W elcom e To MIJA N et Seaton Hall University Portal Internet W ireless Gateway

32 Floor & Building Level Architecture of WSN Roof top Gateway Wireless Gateway 8th th Routers In Secure Locations 6th End Points On Extinguishers Etc.

33 Service Chain based on WSN Solution Internet Wireless Gateway Signaling Empty Notification, , Phone, Page Etc MIJA Data system Auto Signal to NAFED Repair Partner Hotel

34 Example of a WSN in a Vineyard (1)

35 Example of a WSN in a Vineyard (2)

36 Example of a WSN in a Vineyard (3)