CUIML: A Language For the Generation of Multimodal Human-Computer Interfaces

Transcription

1 CUIML: A Language For the Generation of Multimodal Human-Computer Interfaces Christian Sandor sandor@cs.tum.edu Technische Universität München Chair for Applied Software Engineering

2 Abstract DWARF Project at the Technische Universität München UIML complied with some of our requirements Extension of UIML was developed and will now be presented 2

3 What is DWARF? Distributed Wearable Augmented Reality Framework DWARF movie Augmented Reality Wearable Computer 3

4 The Problem Framework for Presentation Layer of Wearable Computers An application should be platform independent 4

5 The Problem Framework for Presentation Layer of Wearable Computers An application should be platform independent Interaction with the user is multimodal Voice Input Actions trigger Mouse Input Actions 4

6 The Problem Framework for Presentation Layer of Wearable Computers An application should be platform independent Interaction with the user is multimodal Voice Input Actions trigger Actions Mouse Input 5

7 CUIML a solution Cooperative User Interfaces Markup Language Based on UIML Separation of document structure and presentation Transformation to markup languages Display in browsers for VRML, VoiceXML, HTML (Views) New concepts: Manipulators 6

8 System Design Views UIML: Adresses requirement of platform independence No support for multiple cooperating Views UIML document Generic Structure A B C D Mapping HTML A = H1 D = H2 B = H2 E = H2 C = H3 E Generation Process 1. H2 HTML H1 H3 H2 H2 WML A = W1 D = W7 B = W2 E = W7 C = W2 2. W2 WML W1 W2 W7 W7 7

9 System Design Views (2) Multimodal Human-Computer Interfaces consist of multiple Views B E CUIML document Generic Structure A F C G H D Mapping 1. VRML HTML VoiceXML B = 1 E = 2 2. VRML WML C = 1 F = 2 G = 3 C = 3 F = 4 G = 5 D = 4 H = 5 I = 6 D = 6 H = 7 I = 8 B = 7 E = 8 I Generation Process HCI 1 VRML HTML VoiceXML HCI 2 VRML WML 7 8 8

10 System Design A central component is needed Describes state of the HCI Synchronizes views View View?? View 9

11 System Design A central component is needed Describes state of the HCI Synchronizes views Deterministic Finite Automaton with XML based Configuration View View DFA View 10

12 System Design (2) CUIML extended with B E CUIML document Config = Generic Structure A F C G H D Mapping 1. VRML HTML VoiceXML B = 1 E = 2 2. VRML WML C = 1 F = 2 G = 3 C = 3 F = 4 G = 5 D = 4 H = 5 I = 6 D = 6 H = 7 I = 8 B = 7 E = 8 I Generation Process VRML 1 2 VRML HCI 1 VoiceXML HCI 2 HTML WML

13 Example State transitions of a View Printer available PrintJobQuestion Yes No Confirmed Rejected VRML View 12

14 Example State transitions of a View Printer available PrintJobQuestion Yes No Confirmed Rejected VRML View 13

15 Example State transitions of a View Printer available PrintJobQuestion Yes No Confirmed Rejected VRML View 14

16 System Design Manipulators Views have to be modified by the Access mechanisms for Views differ View?? 15

17 System Design Manipulators Views have to be modified by the Access mechanisms for Views differ Every View needs an appropriate Manipulator Described in behaviour section View Manipulator 16

18 System Design Manipulators ( Example of a Manipulator CUIML document Structure A B C D E Generation Process Manipulator A Behaviour Event "X" Event "X" B C D E A=> B C D E Manipulator A B C D E 17

19 Benefits of CUIML Eases the task of developing multimodal HCIs Dynamic reconfiguration of the HCI to preserve functionality Incorperates the advantages of UIML Separation of development tasks Platform independence 18

20 Future Work Rewrite renderers with JavaCC Adapt the Human-Computer Interface at runtime 19

21 Future Work Rewrite renderers with JavaCC Adapt the Human-Computer Interface at runtime Establish standard metaphors