A Platform for Output Dialogic Strategies in Natural Multimodal Dialogue Systems. Meriam Horchani, Laurence Nigay and Franck Panaget

Size: px

Start display at page:

Download "A Platform for Output Dialogic Strategies in Natural Multimodal Dialogue Systems. Meriam Horchani, Laurence Nigay and Franck Panaget"

Oscar Derek Lucas
5 years ago
Views:

1 A Platform for Output Dialogic Strategies in Natural Multimodal Dialogue Systems Meriam Horchani, Laurence Nigay and Franck Panaget Presentation by Paul Freitas Motivation Multimodal Communication Very natural Example: This presentation Modalities used: Speech Text Graphics Gestures Others? Primary modality: speech Others are complimentary or redundant

2 Motivation Example, Cont. Dialogic Strategy Assignment of content to modalities Selection Process: Based on: Content to present Presentation constraints Audience expectations Real-time evaluation of audience reactions Probably more Complicated, and not well Focus of this paper Directory system Multimodal input and output This paper focuses on output capabilities Design Goals Enable research into dialogic strategies Rapid development & deployment Examine mutual influence of content and presentation Example: issues dealing with number of results Integrates multiple systems & architectures

3 Coming up Multimodal Output s s Dialogic Strategy Component Related and Future Work Multimodal Output: The CARE Properties Nigay and Coutaz, 1997 Describe interaction possibilities of modalities A set M of modalities may be: Complementary-if all modalities in M may be used to complete a task, but no subset of M is satisfactory Assigned (to a particular task)-if the task may not be completed by any other set of modalities Redundant-if they are equivalent and may be used concurrently Equivalent-if they may each accomplish the task alone These definitions are used as the basis for realizing a dialogic strategy

4 Multimodal Output: the Reference Model Bordegoni et al., Layers Control flow not linear All layers receive input from the application s expert systems The Reference Model, Cont. Control Selects the next high-level goal from a list Content Refines goal into sub-goals Selects necessary content to convey Selects modalities to use Design Layout Presentation planning within the constraints of each modality Realization Produces a concrete specification of the design Presentation Display and coordination

5 Architecture Based on ARCH Generic Interactive system architecture Separates Domain-specific logic, dialogue management, and presentation Authors add a Dialogic Strategy Component ARTIMIS and ICARE used to realize the components ARTIMIS Sadek et al Natural Dialogue system Based on a formal theory of interaction Rational behavior Communication Cooperation Supports both input and output of natural language

6 ARTIMIS Natural language output subsystem Linguistic Act Planner Turns intentions of the system into communicative acts Linguistic Realizer (not used As part of the RM: Control layer Part of content layer As part of ARCH: Domain-Specific component Domain-Adapter component Dialogue component ICARE (Interaction-CARE) Bouchet, Nigay and Ganille, 2004 Multimodal interface tool Specification Development Based on CARE Properties Original work focuses on input Holds for output as well

ICARE Specification Elementary components Device components Abstracts device controls and constraints Language components Abstraction of communication events Composition components Describe combined

7 ICARE Specification Elementary components Device components Abstracts device controls and constraints Language components Abstraction of communication events Composition components Describe combined usage of modalities Based on CARE Properties ICARE Specification Example Basic case No user preferences No presentation constraints Complementarity Defines a single communicative act consisting of multiple basic acts

8 ICARE Specification, Continued User preference constraints added ICARE Specification, Final Presentation constraints added

9 The Dialogic Strategy Component Other portion of the Content layer New addition to ARCH diagram Mediator The Dialogic Strategy Component (Cont) Uses the JADE Semantic Agent (JSA) framework Rules engine for communication between intelligent agents Determines response to messages from: Dialogue Component Presentation Component Response may be to either component Seems to overlap with ICARE specifications

10 Related Work Roth and Hefley, 1993 Architecture for Intelligent Multimedia Presentation Systems (IIMPS) Also based on ARCH Concept only Proposes coordination between content selection and presentation design Rousseau et al., 2006 What, Which, How Then (WWHT) Similar goals Focused on automatic adaptation and evolution Content not influenced by presentation Related Work, cont. Zhou et al., 2005 Responsive Information Architect (RIA) Allocates media to communicative tasks Uses an optimization-based graph matching to map data to media (NP-complete) Weighting based on: Data-media compatibility Presentation consistency constraints Uses probabilistic algorithm Does not have the same link between presentation and content

11 References Bordegoni, M. Faconti, G., Feiner, S., MayBury, M. T., Rist, T., Ruggieri, S., Trahanias, P. and Wilson, M. A standard reference model for intelligent multimedia presentation Systems. Computers standards and interfaces 18, 6-7 (Dec. 1997), pp Bouchet, J., Nigay, L., and Ganille, T. ICARE Software Components for Rapidly Developing Multimodal Interfaces Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., Young, R. M. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The Care Properties. Proc. Interact, pp Louis, V. and Martinez, T. The JADE Semantic Agent: Towards Agent Communication Oriented Middleware. AgentLink News 18, August pp LR.pdf#page=16 Sadek, D. Design considerations on dialogue systems: from theory to technology the case of ARTIMIS. Proc. IDS 1999, pp Sadek, M.D., Bretier, P., and Panaget, F. ARTIMIS: Natural Dialogue meets Rational Agency. Proceeding of IJCAI, 1997 Discussion? Discussion!

Human-Computer Interaction (HCI) Interaction modality and Multimodality Laurence Nigay

Human-Computer Interaction (HCI) Interaction modality and Multimodality Laurence Nigay University of Grenoble CLIPS-IMAG Laboratory User Interface Engineering Team Introduction: the Domain Human-Computer