A Dialog Model for Multi Device Interfaces with Different Modalities

Transcription

1 A Dialog Model for Multi Device Interfaces with Different Modalities Robbie Schaefer, Wolfgang Mueller Paderborn University Fuerstenallee 11 D Paderborn, Germany Steffen Bleul University of Kassel Wilhelmshoeher Allee 73 D Kassel, Germany Abstract As a variety of interaction devices are available that support manifold modalities and capabilities, the provision of user interfaces for all these devices becomes a challenging task. This lead to a renaissance of model based UI design in the last few years, as the ability to provide a model or a set of models for many interfaces is a key requirement for context dependent multimodal and multi-device interaction, which avoids multiple implementations for each device or modality. To this end we present a dialog model (MIPIM) that is suited for multi device interfaces and can be adapted towards different combinations of context, device, and user profiles. 1 Introduction Mobile devices and disappearing computers raise new challenges in UI and network technologies in the emerging field of Ambient Intelligence (Aarts, 2002), which builds on the three key technologies Ubiquitous Computing (Weiser, 1991), Ubiquitous Communication, and Intelligent User Interfaces. To integrate different devices requiring multiple multimodal user interactions for AmI environments, we introduce our Multimodal Interface Presentation and Interaction Model (MIPIM). MIPIM is a novel model based dialog model for the specification, provision, and automatic adaptation of multimodal User Interfaces. During the runtime of a UI, the Multimodal Interaction component accepts Input Events in different modalities and triggers the Behavior Resolver, which in turn starts generating the respective UI presented by the Multimodal Presentation component with the activated modalities. The UI is defined by DISL (Dialog and Interaction Specification Language), which combines concepts of UIML (Abrams et al., 1999) and DSN (Curry & Monk, 1995) clearly separating structure, style, and state based behavior in the specification of an interactive dialog. For multimodal interaction, the presentation component maps generic widgets to widgets in a specific modality, while the interaction component does a reverse mapping of these widgets and by that allowing the use of basically any device or modality for interaction. The remainder of this article is organized as follows. After a short review of related work, we present essentials of our MIPIM dialog model. The implementation into a concrete language based on UIML is sketched in the section thereafter, with a focus on context dependent provision and modification of UIs. The article finally closes with conclusions and an outlook to our further research.

2 2 Related Works The W3C has introduced the MMI (Multimodal Interaction) to integrate multimodal user interactions (Larson, 2002). In that context, several XML based languages are under application, like SMIL, EMMA, VoiceXML, InkXML, and UIML. However, they all are dedicated for just one specific modality (e.g., speech, stylus, graphic bases UIs). UIML, VoiceXML, EMMA, and SMIL provide partial support for user interaction description. Nevertheless, they are still rather limited for the specification of more complex state based user dialogs. For dialog specification there exists several classical approaches, which focus on the description of more complex user dialogs interactions for Graphical User Interfaces like Dialogue-Nets (Janssen, 1993), Petri-Nets (Roche et al., 1999), UAN (User Action Notation) (Hartson et al., 1990), and ODSN (Object- Oriented Dialog Specification Notation) (Szwillus, 1997). They all refer to the same basic concepts of parallel Finite State Machines and mainly differ in their description means and hierarchical decomposition into components. Therein, user dialogs are defined by means of states and state transitions, which are triggered by events from user interface elements. However, they do not give a sufficiently tight integration with the UI style and structure and do not consider multimodal applications and adaptations, respectively. Most recently, model based approaches were introduced by (Eisenstein et al., 2001) and (Paternò & Santoro, 2002) to support the isolation of device specific features for facilitating UI adaptations in the context of mobile devices. Our MIPIM model is comparable to the latter model based approaches. In contrast to them, it considers lower levels of details by the means of DISL specifications, where DISL gives a tight integration of modality independent specification of UI style, structures, and complex dialogs. 3 Dialog Modeling for Multi Device Interfaces Initially, our dialog model has been designed to support multi device interaction and rendering on devices with limited capabilities such as mobile phones. This includes different interaction modalities like speech, GUIs, and simple gestures. We see a dialog as a cyclic process where after each completed cycle, a new interaction device can be used to trigger events as depicted in Figure 1. This corresponds to the definition of supplementary modalities, as defined in the multimodal interaction requirements of W3Cs Multimodal Interaction Framework (Maes & Saraswat, 2003): supplementary use of modalities describes multimodal applications in which every interaction (input or output) can be carried through in each modality as if it was the only available modality. To use different modalities with one dialog model, our concepts are based on abstract widgets due to the work of (Plomp & Mayora-Ibarra, 2002), which supports the creation of a vocabulary of very simple UI elements that are modality agnostic and can be used for example with speech, gestures (does not allow direct text processing however) and graphical user interfaces (Mueller, Bleul & Schaefer, 2004). Note that the term widget refers to an abstract user interaction entity and does not refer only to graphical user interfaces rather. We take it as the smallest denominator for all supported modalities. A typical dialog flow as seen in Figure 1 starts with the presentation of the interface on the activated modalities or devices (1). The most important step is the mapping of abstract to concrete widgets, which is done through additional structure- and style definitions that are read for each modality / device. After sending this concrete specification to a client device or by using a transcoded format, the client waits for user inputs (2). Once an input event occurs it is processed on the client provided the client is capable to process it. Otherwise, the input event is passed to the

3 MM Interaction component (3). This component executes a reverse mapping of the concrete widget to an abstract widget - again through the structure and style description - and passes the result to the Behavior Resolver (4), which contains the processor for the control model. Figure 1: Interaction Cycle The control model is based on the Dialog Specification Notation (Curry & Monk, 1995), which is well suited to deal with concurrent states that often occur in user interfaces rather than other solutions, which just implement deterministic finite state machines. DSN allows the bundling of several local UI states and performs multi-state transitions without the need of specifying each transition. We make this possible through the definitions of variables, which capture UI states and events together with rules that map the cross product of multiple states and events to a new set of states in one pass. The rules consist of a condition and a body, where the condition may range over sets of states and sets of events. The body executes when the specified events occur and the entity is in the specified state. In that case, the condition is true and the execution of the body may perform a state transition, which again operates on the state space. Consider the following simple rule in DSN example, which is part of a volume control for a music playing application: USER INPUT EVENTS switches (ivolumeup, ivolumedown, iplay, istop) SYSTEM STATES volume (#loud #normal #quiet) power (#on #off) RULES #normal #on ivolumeup --> #loud The rule defines four user input events and two system states. It can only fire if the volume of the object is set to #normal, power is #on and the ivolumeup switch has been operated. After firing, the rule sets volume to #loud. 4 Context Based User Interfaces For the definition of context based user interfaces, we have defined DISL (Dialog and Interface Specification Language), which implements our DSN-inspired dialog model for multi device interfaces is based on UIML (Abrams, Phanouriou, Batongbacal, Williams & Shuster, 1999). To

4 keep track of the application state within the UI-description, which is a key requirement to allow multi device interaction, we extended the behavioral part of UIML with advanced means to specify rules and the concept of real variables holding various system and UI states. For outlining the changes of UIML s behavior description, we pick up the volume control example again. The DISL behavior section for the previously given DSN example looks as follows: <behavior> <variable id="volume" internal="no" type="integer">128</variable> <variable id="incvolumevalue" internal="no" type="integer">20</variable> <rule id="incvolume"> <condition> <equal> <property-content generic-widget="incvolume" id="selected"/> <value-of>yes<value-of> </equal> </condition> </rule> <transition> <if-true rule-id="incvolume"/> <action> <statement assignment="add"> <variable-content id="volume"/> <variable-content id="incvolumevalue"/> </statement> <statement> <property-content id="visible" generic-widget="apply">yes</property-content> </statement> </action> </transition> <behavior> First, variables for the current volume and a value for increasing the volume are assigned. The rule "IncVolume" implements the condition that evaluates to true, if the widget "IncVolume" is selected. After conditions of each rule are evaluated, we have to decide which transitions have to fire. This is done for every transition, where the condition of the if-true tag is true, and then a set of statements is processed in the action part. There, the "incvolumevalue" is added to the previous set volume, and statements update the UI, e.g., setting a "yes" and "cancel" control. Such variables also allow to process context information in addition to ordinary system and user events. Then, for applications in private environments, we can use several secured profiles for retrieving the current situation in order to dynamically adapt user interfaces. This profiled information may influence specific user interface properties at runtime, when the status of variables changes due to changes in the environment or modified user profiles. On the other hand it may reflect on the provision of user interfaces to different devices or modalities through an adaptation process that comes before rendering the UI, as described in (Schaefer, Dangberg & Mueller, 2002). 5 Conclusion and Future Work We presented a dialog model for multi device interaction. We introduced abstract widgets that can be mapped to different modalities and through a control model based on DSN, which supports modeling of concurrent states in more complex interfaces. Both concepts are integrated into a UI

5 description language based on UIML (Abrams et al., 1999). We have implemented proof of concept demonstrators for a mobile phone, a wireless sensor device for simple gestures, and for simple voice dialogs. Our observation was, however, that for true multimodality support, i.e., including complementary modalities, DISL needs additional features to better orchestrate the different modalities, i.e., for advanced temporal behavior. As DISL is prepared for profile based context handling, we currently focus on their integration. For the validation of applications in that context, we still need more studies with sufficiently complex profiles, which are currently developed in the UBISEC project. Acknowledgements The work described herein was funded through the UBISEC project IST References Aarts, E. (2002). Ambient Intelligence in Homelab, Royal Philips Electronics. Abrams, M., Phanouriou, C., Batongbacal, A.L., Williams, S.M., & Shuster, J.E. (1999). UIML: An Appliance-Independent XML UI Language. Computer Networks 31, Elsevier Science. Bulterman, D., Grassel, G., Jansen, J., Koivisto, A., Layaida, N., Michel, T., Mullender, S. & Zucker, D. (2005). Synchronized Multimedia Integration Language (SMIL 2.1). W3C Working Draft 01 February Chou, W., Dahl, D.A., Johnston, M., Pieraccini, R., & Raggett, D. (2004). EMMA Extensible MultiModal Annotation Markup Language. W3C Working Draft 14 December Curry, M.B., & Monk, A.F. (1995). Dialogue Modeling of Graphical User Interfaces with a Production System. Behavior and Information Technology 14(1): Eisenstein, J., Vanderdonckt, J., Puerta, A. (2001). Applying Model-Based Techniques to the Development of UIs for Mobile Computers. In: Proc. of Intelligent User Interfaces Conference. Hartson, H.R. et al. UAN: A User-Oriented Pepresentation for Direct Manipulation Interface Designs. In ACM Transactions on Information Systems, 8( 3), July Janssen, C. Dialogue nets for the Description of Dialogue Flows in Graphical Interactive Systems. In Proceedings of Software-Ergonomie 93, Teubner, Stuttgart, Larson J.A.; and Raman T.V. (2002). W3C Multi-Modal Interaction Framework. W3C Note. Mueller, W., Schaefer, R., & Bleul, S. (2004). Interactive Multimodal User Interfaces for Mobile Devices. In: Proceedings of 37 th HICSS, Waikoloa HI, USA, Computer Society Press. Maes, S. H., & Saraswat, V. (2003). Multimodal Interaction Requirements.W3C Note, Jan Paternò, F., Santoro, C. (2002). One Model, Many Interfaces. In: Proc. Fourth International Conference on Computer Aided Design of User Interfaces (CADUI2002), Kluwer. Plomp, J., & Mayora-Ibarra, O. (2002). A Generic Widget Vocabulary for the Generation of Graphical and Speech-Driven User Interfaces. In: Int. Journal of Speech Technology 5, Roche, P., d Ausbourg, B., Durrieu, G.. Deriving a Formal Model from UIL Description in Order to Verify and Test its Behavior. In: Proc. of the 3rd Eurographics Workshop on Design, Specification, and Verification of Interactive Systems, Schaefer, R., Dangberg, A., & Mueller, W. (2002). Fuzzy Rules for HTML Transcoding. In: Proc. of 35 th HICSS, Maui HI, USA, Computer Society Press. Szwillus, G.. Object Oriented Dialogue Specification with OSDN. In: Proc. of Software- Ergonomie 93, Teubner, Stuttgart, Weiser, M. (1991). The Computer for the 21st Century. Scientific American 265(3):