Standards for Multimodal Interaction: Activities in the W3C Multimodal Interaction Working Group. Group

Transcription

1 Standards for Multimodal Interaction: Activities in the W3C Multimodal Interaction Working Group Deborah A. Dahl Chair, W3C Multimodal Interaction Working Group

2 Overview Goals of the W3C MMI group MMI use cases MMI requirements MMI specifications Schedule

3 Goals Develop standards for multimodal applications used on the World Wide Web Accommodate both current and future modalities Support local, server-based, or distributed applications Support multiple devices

4 Process Analyze use Cases examine a wide variety of current and future use cases Develop and publish requirements cover current needs, provide for extensibility to new modes, leverage other web standards Publish specifications -- meet requirements, address needs of platform and developer communities, support rich and compelling multimodal applications

5 Use Cases: GUI, Ink, Voice Show me all the restaurants in this area.

6 Use Cases: GUI, Pointing, Voice Show me all the trails starting near here.

7 Use Cases: Driving Directions User: I want to go to the Hotel del Coronado. System System looks looks up up route route and and finds finds out out about about traffic traffic conditions conditions System: There's heavy traffic on the normal route, so I'll give you an alternate route. It s about 8 miles. User: Show me the route on the map.

8 System: Now turn left onto North Harbor Dr. You'll be going 1.4 miles User: Could you speak a little louder? System: Ok. Should I repeat the last thing I said? User: No thanks. Car Car starts starts moving, moving, GPS GPS input input gives gives the the system system the the location location of of the the car car

9 User: What street am looking for again? System: The next turn will be a left onto West Grape Street. User: How much farther to the next turn? System: You have about one mile to go. System: Merge onto California 75 Coronado Bay Bridge. System System answers answers questions questions about about current current status. status.

10 User: Is there a pizza place near here? System: The nearest pizza place is Domino's at 1330 Orange Avenue,.4 miles away. User: I want to stop there. System: Ok. user user starts starts getting getting hungry hungry user user is is routed routed to to pizza pizza place, place, eats eats pizza, pizza, and and resumes resumes driving driving to to hotel hotel

11 Multimodal Requirements Selecting/Constraining available modes Allowing user to select which mode are used by user/system Allowing system to constrain modes (road safety) Flexible use of output media Using ouput media in a complementary fashion Using output media for redundancy Using media in a sequential fashion Flexible use of input modes Allow input in choice of modes (voice or stylus/keypad) Allow for inputs combining more than one mode 'I want this one' while drawing circle on picture Resolve inconsistent inputs Saying yes while typing no

12 Multimodal Requirements Support a wide range of device capabilities Allow for multiple users/devices/servers Assisting communication between human users Local or Network based processing Allow integrated and distributed architectures Allow device to change as user s environment changes Extensible to new modes as they become available

13 MMI Specifications Framework specification Describes how modalities are integrated into multimodal applications EMMA (Extensible MultiModal Annotation) defines format for results returned by a recognizer ink markup language specification for capturing pen/stylus input

14 EMMA (Extensible MultiModal Annotation) Represents the normalized result of processing a user input text ink speech Includes metadata about timestamps, confidences, simultaneous inputs from multiple modalities, etc. A data model San Diego <destination> san diego </destination>

15 Ink Input Uses Gestures Drawings Note taking Handwriting recognition Signature verification Specialized notations: Math, Music, Chemistry,... Why develop a standard ink format? No existing format meets requirements XML based format InkXML contribution (IBM, Intel, Motorola, International Unipen Foundation)

16 Current Document Schedule Multimodal Interaction Framework First Working Draft - June 2003 Last Call Working Draft - December 2003 Extensible Multimodal Annotation Markup Language (EMMA) First Working Draft -- April 2003 Last Call September, 2003 Pen input First Working Draft -- April, 2003 Last Call September, 2003

17 Submissions and Contributions Multimodal Approaches XHTML + Voice submitted by IBM, Motorola, Opera SALT contributed by the SALT Forum Ink Approaches InkXML contributed by IBM, Intel, the International Unipen Foundation, and Motorola

18 Find out More Multimodal Interaction Working Group home page: W3C Home Page:

19 Published Documents Framework: Use Cases: MMI Requirements: EMMA Requirements: ink Requirements:

20 Acknowledgements Thanks to Dave Raggett, W3C Staff contact for MMI and Voice, for help with the slide material.