SMUIML Editor. Graphical Tool for Modeling Multimodal Interaction. Master Thesis. Saïd Mechkour

Transcription

1 SMUIML Editor Graphical Tool for Modeling Multimodal Interaction Master Thesis Saïd Mechkour 08 th September 2011 Thesis supervisors: Dr. Bruno Dumas Dr. Denis Lalanne and Prof. Dr. Rolf Ingold DIVA Group Document, Image and Voice Analysis Group Department of Informatics University of Fribourg (Switzerland)

2

3 I dedicate this work to my beloved family especially my parents,and my wife.

4

5 Acknowledgements I would like to express my sincere gratitude to everyone who supported me during this project. To Prof. Dr. Rolf Ingold, the director of my thesis, for giving me the opportunity of achieving my thesis in Document, Image and Voice Analysis Group. To Dr. Bruno Dumas, my technical supervisor, for introducing me to HephaisTK project, his feedbacks, and also for extending HephaisTK especially for this work. To Dr. Denis Lalanne, my supervisor, for his help in countless scientific discussions, his feedbacks, and his valuable suggestions. To Chen Kai, and Matthias Schwaller for participating to the usability test and providing valuable suggestions. A special thanks goes to those who, besides Denis Lalanne and Bruno Dumas, have proofread this dissertation: Anna Bloch Mechkour and Adriana Wilde. Finally, to my colleagues: Adriana Wilde, Chen Kai, Aron Martinez, Matthias Schwaller, and Cavadini Daniele, for friendship, help and encouragement throughout the time of my M.Sc. study. iii

6

7 Abstract The goal of this thesis was to develop a visual graphical user interface to create multimodal interfaces. This interface uses HephaisTK, a toolkit aiming at simplifying the creation of multimodal interfaces (developed during the Ph.D. thesis of B. Dumas at the University of Fribourg). This toolkit uses a configuration file (SMUIML file) to support the creation of multimodal human-computer dialog scenarios for multimodal applications. The SMUIML configuration file is written in XML. To ease the development of multimodal applications with HephaisTK, this master thesis aims at designing, implementing and evaluating a visual editor of SMUIML file. In order to edit visually this file we have chosen visual and interaction techniques used to deal with nested and highly structured information. For this reason, special attention was given to the design and graphical representation of SMUIML. This master thesis report first describes the various technologies used to develop the SMUIML Editor. It then describes how the SMUIML editor was implemented. Additionally, an informal user study is presented with two different applications that use HephaisTK, with the goal to demonstrate that the SMUIML Editor is fully functional and meets the objectives of this work. At the end of this work, we can happily say that the SMUIML Editor is fully functional and reduces the complexity of developing multimodal application based on HephaisTK. Keywords: Multimodal interfaces creation, graphical editor, eclipse platform, HephaisTK, SMUIML. v

8

9 Table of Contents 1. Introduction Context and Motivation Objectives Structure of Thesis Background and Context Multimodal interfaces Definition HephaisTK Synchronized Multimodal User Interaction Modeling Language (SMUIML) State of The Art Visual Programming System Survey of Three Visual Programing Systems Techniques and Approaches suited for SMUIML Editor Technologies Chosen Technologies Eclipse Eclipse Platform vs Eclipse SDK Platform Runtime and Plug-in Architecture The Graphical Editing Framework (GEF) Graphical Modeling Framework (GEF) architecture Eclipse Modeling Framework (EMF) EMF Models EMF Code Generator Design of SMUIML Editor Problem Analysis vii

10 5.2. Problem Solution Graphical Representation of SMUIML Visual Edition of SMUIML Presentation of SMUIML Editor Views of SMUIML Editor Actions of SMUIML Editor Monitoring Tool Implementation SMUIML Editor Architecture EMF Model Specification Generation of a Basic Editor as Plug-in Model Generation Generation of a Basic Editor The EMF Generated Packages Visual Editor Page Implementation Model View Controller Integrating with the workbench Dialog Page Packages Adaptation of Generated Code Validation Validation with Users Results from Involving Development Participants User Study SMUIML Editor Evaluation First Use Case Second Use Case Results and Conclusions Conclusions and Possible Future Work Contributions Future Work A. SMUIML XML Schema 51 B. Setup SMUIML Editor 55 B.1. Prerequisites B.2. Plug-in Installation B.3. Getting Start

11 C. Common Acronyms 59 D. CD-ROM 61 References 63

12

13 List of Figures 2.1. Multimodal interaction, from [6] Mutlimodal Application developed from scratch Mutlimodal Application developed upon HephaisTK Architecture of HephaisTK, from [7] Relationship between the three levels architecture and main sections of SMUIML, from [7] Mains components of the Eclipse Platform, from [14] Eclipse Plug-in Architecture GEF Model View Controller (MVC) architecture, from [20] Relation between Ecore model and other models sources, from [8] Editing SMUIML configuration file and Java handler cause user frustration Relationships among elements in SMUIML file Java handler is generated from SMUIML configuration file which is created through Graphical User Interface (GUI) Numbers from 1 to 4 are steps of designing Visual Programing Language (VPL) of Visual Programing System (VPS) that generate SMUIML, while 5 represents an equivalent dialog element written in SMUIML General overview of SMUIML Editor Partial part of SMUIML Editor toolbar Selection of state start by monitoring tool Selection of state state1 by monitoring tool Selection of state state2 by monitoring tool High level architecture representation of SMUIML Editor Serialization of an Ecore EClass and Attribute are defined by the value of the attribute name of EAnnotation extendedmetadata. Here, Actions are serialized as actions UML representation of SMUIML model generated from Ecore model using the Ecore diagram tool xi

14 6.4. From XML Schema to an EObject model Architecture of generated editor The EMF generated packages and their contents Architecture of diagram page of SMUIML Editor Modification of Ecore Not reachable states from state start are highlighted with yellow icons The state in the right shows how SMUIML highlight an error How layout and zoom help users to visualize the graph Multimodal Dialog Senario used by contact manager application Multimodal Dialog Senario used by Exposé application B.1. How to access the installation wizard of Eclipse B.2. Select all available sites in first page of installation wizard B.3. The red arrow shows where you can find the current open perspective B.4. How to access to the menu items SMUIML File and HephaisTK Project. 57 D.1. Tree view of the content of the CD-ROM D.2. The CD-ROM of this project

15 Listings 2.1. handler.java: A Java handler smuiml.xml: SMUIML template SmuimlResourceFactoryImpl.javal: SMUIML template A.1. smuiml.xsd: SMUIML template xiii

16 1 Introduction 1.1. Context and Motivation Multimodal interfaces are a particular kind of interfaces for human-computer interaction, which allow users to communicate with computer systems using different modes of interaction. To facilitate development of such interfaces many frameworks and tools were created. The Hephais Tool Kit (HephaisTK) is one of these tools developed by Dumas during his Ph.D. thesis at the University of Fribourg [1]. This master thesis takes place in the context of the HephaisTK project which aims to provide a toolkit that simplifies creation of multimodal interfaces. The integration of this toolkit in developers applications is made possible via a configuration file. Such file is written in a domain-specific language built on XML and named Synchronized Multimodal User Interaction Modeling Language (SMUIML). HephaisTK provides a prototyping tool to develop multimodal applications. It does so by allowing developers to focus on the design of multimodal human-machine dialog scenarios for their application. The multimodal human-machine dialog scenario is created with a domain-specific language called SMUIML and then used as configuration file by HephaisTK. SMUIML was designed to create configuration files that are expressive and easy to understand by developer without extensive knowledge about multimodal interaction. A SMUIML file facilitates the integration of HephaisTK and the client application. However, its usability is very limited. Critically, any newcomer would have a steep learning curve before creating a SMUIML file. Dumas et al. (2010) [2] have shown that SMUIML is easy to understand due to choice made by the authors to balance between usability and expressiveness. Usability is addressed from a human point of view and refers to how a language is easy to learn, remember and use, while expressiveness is related to more technical features of the programming language such as extensibility, completeness and reusability. Having to use a simple text or a standard XML editor to create and edit SMUIML file is the main reason behind the lack of usability of HephaisTK. This lack of usability is compounded by to the following disadvantages: no visual feedback to highlight existing relationships among elements; 1

17 2 Chapter 1. Introduction difficulty of producing mental representation of the dialog scenario from the configuration file; tasks of creation, edition and visualization of SMUIML file are time consuming; dynamic error checking; 1.2. Objectives The main objective of this thesis is to facilitate the development of multimodal interfaces that use the framework HephaisTK. For this reason, a GUI needs to be created, allowing developers: to model multimodal interaction of their application visually, and generate the corresponding SMUIML file as well as Java handler; to execute the multimodal client application based on HephaisTK; 1.3. Structure of Thesis The remainder of this thesis is structured as follows: Chapter 2 gives an overview of the context and the background of this thesis. Chapter 3 concerns the state of the art of visual programming environments. Chapter 4 describes the technologies used and justifies their choice. Chapter 5 discusses our approach in order to meet the objectives of this thesis. Chapter 6 gives an overview of how the SMUIML Editor was implemented and describes its various packages. Chapter 7 shows how SMUIML Editor was validated. Finally, the conclusions of this thesis are presented in Chapter 8.

18 2 Background and Context 2.1. Multimodal interfaces Definition HephaisTK Synchronized Multimodal User Interaction Modeling Language (SMUIML) This chapter aims to give an overview of multimodal interfaces, HephaisTK and SMUIML. Multimodal interfaces are part of the general background of this thesis while HephaisTK and SMUIML are its particular context Multimodal interfaces Nowadays computers are becoming more ubiquitous and thus part of everyday life, most particularly in industrialized countries. However, human-computer interaction is not intuitive and still needs great levels of human attention. To make human-computer interaction more natural, some research in this area aims to integrate natural gestures, activity and behavior into human-computer interaction design. In particular, there is a great interest towards multimodal human-computer interaction as means to such integration [3] Definition Multimodal interfaces are interfaces for Human-Computer Interaction (HCI) that use gesture, voice and other modalities as input and/or output. The development of the field of Multimodal Interaction has been possible thanks to advances within multidisciplinary research domains, such as cognitive psychology and computer science. Such interdisciplinary research has relied on theoretical frameworks to help convergence of research. One of the most important theoretical framework of HCI is known as the human action cycle which was coined by Donald A. Norman [4]. As multimodal interaction is a sub-branch of HCI, the original human action cycle can be specifically adapted to the needs of this new class of interfaces. Figure 2.1 is an adaptation of Norman s action cycle to multimodal interaction as presented by Dumas et al. in [6]. 3

19 4 Chapter 2. Background and Context Figure 2.1.: Multimodal interaction, from [6]. Input modalities use a variety of input devices and sensors available within computer systems. Amongst the mostly used input modalities are speech, touch, gesture, movement, pressure, acceleration, etc. Multimodal systems are generally designed to support an input that is composed of at least two modalities. Extracting information from composed input and its interpretation is called multimodal fusion. Generally output modalities of multimodal systems require at least one of the five human senses, each of these modalities can be perceived through any human sense. The process of generation an adequate message to the user is called multimodal fission and can be done in dynamic and/or static way. Multimodal fusion and fission are generally not easy to design and are the central problems in multimodal interaction design. Therefore, many researches have attempted to create frameworks that facilitate their design. HephaisTK is a particular example of this approach, which is described in full in the next section HephaisTK HephaisTK is the toolkit used in this project which helps developers of multimodal interfaces to create multimodal applications in an easy way. This is done by separating the process of collecting data from the input sensors and their fusion from the multimodal application. Figure 2.2 shows a multimodal application developed from scratch where input modalities fusion and output modalities fission are hard-coded to meet a specific multimodal dialog scenario of application. Figure 2.3 shows another multimodal application that uses HephaisTK, where the fusion of input modalities is build-up dynamically from SMUIML file. Only the fission of output modalities is hard-coded by the multimodal

20 2.1. Multimodal interfaces 5 application. In the remainder of this thesis we will refer to any multimodal application that uses HephaisTK as client application. Figure 2.2.: Mutlimodal Application developed from scratch. Figure 2.3.: Mutlimodal Application developed upon HephaisTK. Architecture HephaisTK is built on agents architecture in order to run in a distributed environment. The toolkit consists on many kinds of agents according to their roles, as follows. Recognizer agents are responsible of collecting and encapsulating data from recognizers before delivering it to the Postman agent. Postman agent, as the name suggests, receives data

21 6 Chapter 2. Background and Context and distributes it to other agents in the toolkit that is concerned with process of interpretation. The process of interpretation of data collected by Postman agent is done via an integration committee which consists of two types of agents. A first type of agent named FusionManager is responsible for process of fusion which aims to give means to the data collected from different recognizers. A second type of agent helps FusionManager agent to extract means and send notification to the client application. Further details can be found in [7]. Figure 2.4.: Architecture of HephaisTK, from [7]. In order to use HephaisTK developers need to create a configuration file written in SMUIML to describe multimodal dialog scenarios and a Handler file written in Java. SMUIML file is used to create agent of integration committee related to client application defined by attribute client of integration_description element in the same file. Code extract 2.1 is a simple template of the handler, is used here to describe the integration of client application and HephaisTK. Line 11 of Code extract 2.1 shows how HephaisTK is instantiate for a given client application where the parameter client_app refers to the attribute client of the element integration_description of the SMUIML file "smuiml.smuiml". Line 12 sets the absolute path of SMUIML file that is used by HephaisTK manager to localize SMUIML file. Line 15 subscribes client application as listener to HephaisTK fused events. Listeners are a standard construct of the Java programming language. The fusedevent method defined in the line 23 implements the method defined by MultimodalInputFusedEventListener. This method is called each time a fusion event occurs in HephaisTK. Events fused by the HephaisTK s Integration Committee are represented as message values, which are defined in the actions section of a typical SMUIML file.

22 2.1. Multimodal interfaces 7 1 import d i u f. diva. hephaistk. c o n f i g. HephaistkInitManager ; import d i u f. diva. hephaistk. d i a l o g. Message ; 3 import d i u f. diva. hephaistk. events. MultimodalInputFusedEvent ; import d i u f. diva. hephaistk. events. MultimodalInputFusedEventListener ; 5 public class handler implements MultimodalInputFusedEventListener { private s t a t i c f i n a l S t r i n g SMUIML_FILE_NAME = " smuiml. smuiml " ; 7 private s t a t i c f i n a l S t r i n g SE = System. getproperty ( " f i l e. separator " ) ; private s t a t i c f i n a l S t r i n g SMUIML_FILE_PATH = System. getproperty ( " user. d i r " ) + SE + SMUIML_FILE_NAME ; 9 public handler ( ) { / / here comes i n s t a n c i a t i o n of HephaisTK 11 HephaistkInitManager hephaismanager = new HephaistkInitManager ( " c l i e n t _ a p p ", " hephaistk_config. xml " ) ; hephaismanager. addsmuimlconfigfile ( SMUIML_FILE_PATH) ; 13 hephaismanager. startframework ( ) ; / / d e c l a r a t i o n of class as l i s t e n e r s implementer 15 hephaismanager. addmultimodalinputfusedeventlistener ( this ) ; } 17 public s t a t i c void main ( S t r i n g args [ ] ) { new handler ( ) ; 19 } / / as t h i s class implements MultimodalInputFusedEventListener, we have to 21 / / o v e r r i d e t h i s class / 23 public void fusedevent ( MultimodalInputFusedEvent event ) { / / ' events ' are messages such as declared i n the <actions > p a r t of your 25 / / SMUIML s c r i p t for ( Message message : event. getmessages ( ) ) { 27 i f ( message. getmessage ( ). equals ( " mail " ) ) { mail ( message ) ; 29 } else i f ( message. getmessage ( ). equals ( "web" ) ) { web ( message ) ; 31 } else i f ( message. getmessage ( ). equals ( " card " ) ) { card ( message ) ; 33 } else i f ( message. getmessage ( ). equals ( " openweb " ) ) { openweb ( message ) ; 35 } else i f ( message. getmessage ( ). equals ( " sendmail " ) ) { sendmail ( message ) ; 37 } } 39 } private void mail ( Message message ) { 41 } 43 private void web ( Message message ) { 45 } private void openweb ( Message message ) { 47 } 49 private void card ( Message message ) { 51 } private void sendmail ( Message message ) { } } Code extract 2.1: handler.java: A Java handler Synchronized Multimodal User Interaction Modeling Language (SMUIML) SMUIML is a description language for multimodal interaction based on XML technologies. It is used to make transparent the integration of HephaisTK and the client application. The main contributions of this thesis are related to improve the usability of SMUIML. All files written in SMUIML respect the template shown at the Code extract 2.2. This

23 8 Chapter 2. Background and Context template is subdivided into four main sections that correspond to the content of XML element integration_description which are recognizers, triggers, actions and dialog. These sections represent the three levels architecture that SMUIML supports. As we can see in Figure 2.5 the mapping is as follows: input/output level corresponds to content of XML element recognizers; events level corresponds to content of XML elements triggers and actions; dialog level corresponds to content of XML element dialog; Figure 2.5.: Relationship between the three levels architecture and main sections of SMUIML, from [7]. 1 <?xml version= " 1.0 " encoding= "UTF 8"?> <smuiml> 3 <integration_description c l i e n t = " c l i e n t _ a p p " > <recognizers> 5 <! here are defined a v a i l a b l e recognizers and t h e i r m o d a l i t i e s > < / recognizers> 7 <triggers > <! here are defined t r i g g e r s by using defined m o d a l i t i e s > 9 < / triggers> <actions> 11 <! here are defined a c t i o n s t h a t t o o l k i t send to the c l i e n t a p p l i c a t i o n > < / actions> 13 <dialog leadtime=" 1000 " > <! here are defined the c l i e n t a p p l i c a t i o n multimodal d i a l o g > 15 < / dialog> < / integration_ description> 17 < / smuiml> Code extract 2.2: smuiml.xml: SMUIML template Recognizers: This section declares a set of recognizers that should be available for a given multimodal dialog scenario. Each recognizer belongs to a specific modality and has a unique name which is used as general identifier throughout the HephaisTK. Triggers: This section declares a set of triggers based on the available modalities (corresponding to all the modalities of declared recognizers). Triggers are referenced within multimodal dialog scenarios (below) to define under which triggers pattern a transition is selected. A pattern of triggers is a collection of triggers combined using a special elements. These elements are seq_and, seq_or, par_and, and par_and. They are SMUIML representation of CARE properties of multimodal interactive systems which are complementarity, assignment, redundancy, and equivalence.

24 2.1. Multimodal interfaces 9 Actions: This section declares a set of actions. Each action has name and defines message for the target client application and is referenced from multimodal dialog scenario by transition. When a transition is selected all actions referenced from it are sent to their target client application. Dialog: This section describes a multimodal dialog scenario for a multimodal client application based on HephaisTK. The dialog scenario is based on state machine abstraction where a context element represents state. The transition between context is selected when a pattern of triggers matches for a given input data.

25

26 3 State of The Art 3.1. Visual Programming System Survey of Three Visual Programing Systems Techniques and Approaches suited for SMUIML Editor In order to be able to develop the SMUIML Editor, which is a graphical tool for modeling human-computer interaction, a survey of visual programming systems was realized. In this section we first define visual programming systems, then three different examples of hybrid text and visual programming systems are compared. Finally, as an outcome of such a survey, some techniques and approaches suitable for the development of the SMUIML Editor are presented Visual Programming System The field of VPL aims to develop languages that use symbols (images, icons, diagrams) to communicate with computers instead of using textual programming languages. Its development is a result of interdisciplinary study, based mainly on study of pictograms and symbols as communication means, computer graphics, programming languages, and human-computer interaction. Nowadays VPSs are used in many applications that concern both specific and more general programming tasks. VPS consists of two components which are VPL and its tool. VPL refers to graphical elements and their relations while, tool represents a software environment where VPL is created and executed. A survey of VPS, by Dowsan and Boshernitan (2004) in [10] shows that VPS can be classified as follows: Purely visual languages. These are used to create programs consisting of only graphical elements. Programs written in purely visual language are executed in direct manner by processing spacial arrangement of its graphical element. Hybrid text and visual systems. These are used to create program that consists of mixture of visual and textual elements and/or is translated into intermediate textual language before its execution. 11

27 12 Chapter 3. State of The Art Constraint-programming systems. This term refer to systems that use a well known constraints principle in order to guide users to combine and manipulate their visual elements. Programming-by-example systems. These are systems that generate a program to perform the tasks in general cases from a specific case defined by the user. Form-based systems. This term refers to systems that use visualization and programming metaphors that are influenced by spreadsheets. The first two categories represent the two most important VPS because almost all VPS can fall at one of these two systems. The three last categories cannot be considered as real groups of VPS because they do not exclude each other. For example, a XML Schema Editor of Eclipse is an hybrid text visual editor and is also considered also as form based system. One of the objectives of this project is to develop GUI that generate SMUIML file. Therefore, the GUI that we aim to develop belongs to the category of hybrid text visual system Survey of Three Visual Programing Systems In order to gather information about hybrid text systems that would be suitable to develop SMUIML Editor, we compared three graphical editors belonging to this category. Special interest was given to the techniques of visualization and interaction used in such systems. The particular editors WTP XML Schema Editor, Ecore Diagram Editor and Faces Configuration Editor were chosen for study because they use slightly different techniques and are also developed for Eclipse IDE platform. An important detail, not yet mentioned, is that all of these editors are used for different purposes. This explains why they do not use the same techniques to visualize the graphical representation of the structured data. Concretely, XML Schema Editor is used to create XML Schema, Ecore Diagram Editor is used to visualize and edit Ecore model, while the Faces Configuration Editor is used to complete the JSF web application development process by allowing edition of faces configuration. The next three paragraphs offer more detail about the said editors. First, the XML Schema Editor can be considered as completely different from the two other editors. The reasons are two-fold: firstly, it does not provide a palette tool to create new elements, and secondly, the general overview of the document does not show relationships among its elements. The reason behind the lack of a general graphical overview of relationships among elements can be explained by the fact that XML Schema files are generally bulky and therefore their respective diagrams are usually too large for the canvas space available. To offer visual relationships among elements, the developer needs to explore the XML Schema from the root element by clicking on this element. This action causes the display of the selected element and its children represented by a directed graph. This kind of visualization is called detail-on-demand and offers a partial solution to visualize a large graph. An unwanted side effect of this approach is that while

28 3.3. Techniques and Approaches suited for SMUIML Editor 13 exploring the XML Schema document the user has only access to its immediate context. The tutorial published by Ha (2011) in [16] provides with more details. Second, the Ecore Diagram Editor gives a general overview of relationships among elements by using a directed graph. All elements of the Ecore model are represented in such graph. Generally, in order to visualize the whole graph, the user needs to scroll up and down, or alternatively, a zoom accessible from toolbar or by context menu can be adjusted. The latter solution cannot be considered as appropriate because it gives only an idea of the topology of the graph (i.e. how the diagram looks like) but often the names of the elements in the graph are unreadable, requiring zooming in and out to access them. More details can be found in [14] Finally, the Faces Configuration Editor is used to edit faces configuration which is used by the JavaServer Faces Tools web application [17]. This editor contains many tabs, each being used to achieve a special task. For example, the tab navigation rule represents the navigation diagram, and is used to review and compose navigation rules between JSP pages. To add new page in faces configuration, the user needs to select Page node in tool palette and drop it on the "Navigation" tab which opens dialog window that asks to select JSP File. The main difference between Ecore Diagram Editor and Face Configuration Editor is that Ecore Diagram represents all elements of the Ecore model, while navigation rule tab of the Faces Configuration Editor represents only some of them. This approach has the advantage to display only a high level information which facilitates reviewing and composing navigation rules. Srinivasan s tutorial (2006) [17] gives more details Techniques and Approaches suited for SMUIML Editor The survey of tools presented in section 3.2 was very valuable because it allowed the evaluation of several visual and interaction techniques before their implementation for the SMUIML Editor. Here are some examples: Tab navigation rule of Faces Configuration Editor visualize only high level concepts of faces configuration. The low level concepts are accessible though other tabs. This editor also use a wizard to select JSP page when a new node page is selected and dropped in the canvas. The high level concept of a SMUIML file is represented by the dialog element which is a state machine. In order to create a Trigger and an Action, a wizard is used. Widely recognized works, such as those by Rogers et al. (2002) [18], and Norman (2002) [4] show that design principles represent a set of guidelines that help developers in their design. The most important design principles are visibility, feedback, affordance, mapping, constraint and consistency. For example, all of the explored editors above described use properties view to set value of attributes of selected element. In order to respect consistency which is one of the most important design principle, a proprieties view will

29 14 Chapter 3. State of The Art be used in SMUIML Editor. None of these editors provide an interesting visualization techniques to visualize large graphs. It will be more appropriate to reduce details of bulky nodes, in order to reduce the size of the graph. Then details are displayed only when the user demands. This solution is also known as generalization of the fisheye visualization technique. According to Noik [19] and M.-A.D. Storey et al. [21], this generalization of fisheye technique prevents losing the context related to the user s interest during the exploration of the graph.

30 4 Technologies 4.1. Chosen Technologies Eclipse Eclipse Platform vs Eclipse SDK Platform Runtime and Plug-in Architecture The Graphical Editing Framework (GEF) GEF architecture Eclipse Modeling Framework (EMF) EMF Models EMF Code Generator For the development of a graphical editor for SMUIML, we began by choosing suitable technologies for our purposes. The preference was given to the Eclipse platform which is known by qualities of its graphical editors and its architecture based on plug-ins. The Eclipse frameworks GEF, EMF and GMF appear to be adequate to meet the project requirements, and they will be discussed below Chosen Technologies Generating almost the entire graphical diagram editor for a given structured data by using GMF is appealing, however in the practice this is not simple. The requirements of graphical editor extend beyond merely having a graphical diagram editor to edit and create instance of a model visually. Most of requirements concern usability of the editor. As we will see, GMF is based on EMF and GEF, and adapting a generated editor requires proficient knowledge of all of these technologies, besides Eclipse platform which requires more time than is available within the scope of this work. To reduce this time we decided not to use GMF. Instead, we used EMF to generate Java classes (plug-in) from the EMF Model. These classes are needed, firstly, to build a graphical editor, and secondly, to save an instance of the model as XML. The generated plug-in is then extended to create graphical diagram editor using GEF framework. This approach contributes to ensure that the model managed by the generated editor is indeed a model representing SMUIML. The following sections will provide short descriptions of these frameworks and Eclipse platform. 15

31 16 Chapter 4. Technologies 4.2. Eclipse The Eclipse project was created at the end of 2001 by IBM [14]. In January 2004, the Eclipse Foundation was created in order to support the Eclipse development community, and to ensure independence of Eclipse project from industrials consortium that supported it Eclipse Platform vs Eclipse SDK Figure 4.1 shows that Eclipse SDK includes Eclipse Platform, Java Development Tools (JDT), and Plug-in Developer Environment (PDE). JDT implements the well known Eclipse Java Integrated Development Environment (IDE). PDE provides tools that help to create, develop, test, debug, build and deploy Eclipse plug-ins, features, and so on. The Eclipse platform is an Eclipse subproject for building software development tools. It is free, open source, written in the Java language. The Eclipse platform provides a plug-in architecture that simplifies the development of software. It provides the core frameworks and services upon which plug-in extensions are created. It consists of many components and each component is built in one or more plug-ins. Figure 4.1.: Mains components of the Eclipse Platform, from [14]. The components introduced above are described below. The workbench provides the User Interface (UI) for developer application. The help system provides browsable books. The workspace is where the data model of an application is located. Team features support repository management and team collaboration. The platform runtime is part of Eclipse core. It provides basic platform infrastructure that supports plug-ins management and execution.

32 4.3. The Graphical Editing Framework (GEF) Platform Runtime and Plug-in Architecture The Eclipse architecture is based on plug-ins, in order to provide for software developers an extensible IDE. All Eclipse components are plug-ins, except the core component. A plug-in is the smallest functional component of Eclipse platform and may be related to another plug-in by dependency or extension relationship. On one hand, a dependency relationship is a relation between two roles, dependent plug-in and prerequisite plug-in. Dependent plug-in depends on one or more plug-ins (prerequisite plug-in(s)) that support its functions. On the other hand, an extension relationship is relation between two roles host plug-in and extender plug-in. The extender plug-in, as the name suggests, extends functions of its host plug-ins (see Figure 4.2). Figure 4.2.: Eclipse Plug-in Architecture The Graphical Editing Framework (GEF) The GEF aims to provide technology that helps to implement rich graphical editors and views for the Eclipse Workbench. It is used in this project to create a graphical editor that edit and visualize SMUIML files. The GEF project is a bundle that consists of three components: Draw2d is a layout and rendering toolkit for displaying graphics on an Standard Widget Toolkit (SWT) Canvas. The GEF is an interactive MVC framework built upon SWT and Draw2d and used to create graphical editors for the Eclipse Workbench User Interface (EWUI). Eclipse Visualization Toolkit (Zest) is a visualization toolkit based on Draw2d, which enables implementation of graphical views for the EWUI GEF architecture Graphical editors that use the GEF follow a MVC architecture, which means that components model and its view communicate via a controller component. Controller acts in

33 18 Chapter 4. Technologies Figure 4.3.: GEF MVC architecture, from [20]. the way when changes are made inside one component are reported to another one, which makes synchronization of the model and its view possible. Figure 4.3 shows architecture of GEF which consist of three components, Model, EditParts and Figures. The GEF is model agnostic which means that is possible to use it for any model. EditParts represents the Controller component of MVC while Figures represents the View component Eclipse Modeling Framework (EMF) The Eclipse Modeling Framework (EMF) project started as part of tools project of Eclipse. Since 2002 the EMF has become a subproject of the Eclipse Modeling Project [8]. In this thesis, the EMF is used to generate Java classes representing a model and a basic tree editor. The Eclipse foundation describes EMF in eclipse.org as follows: EMF is a modeling framework and code generation facility for building tools and other applications based on a structured data model. From a model specification described in XML Metadata Interchange (XMI), EMF provides tools and runtime support to produce a set of Java classes for the model, a set of adapter classes that enable viewing and command-based editing of the model, and a basic editor. Models can be specified using annotated Java, XML documents, or modeling tools like Rational Rose, then imported into EMF. Most important of all, EMF provides the foundation for interoperability with other EMF-based tools and applications. [9] EMF Models The EMF has two different models which are Ecore model and generator model (genmodel). Ecore is meta-meta model that is used to describe any created EMF model. It

34 4.4. Eclipse Modeling Framework (EMF) 19 is an Eclipse implementation of Essential Model Object Facility (EMOF) and forms the basis for the tool development in EMF. Ecore can be created from scratch by using eclipse Ecore editor or imported from different models as shown in Figure 4.4 and is serialized in XMI. Genmodel is a decorator model for Ecore. It allows to control and tailor the code generated using EMF by specifying other information that are not available in the Ecore model. For example, genmodel allows to indicate where the code should be generated. Figure 4.4.: Relation between Ecore model and other models sources, from [8] EMF Code Generator The EMF code generator is called EMF Codegen. It generates Java classes from a model written in XMI that is a conformed instance of the Ecore model, e.g., to develop application that manipulates structured data. The starting point will be to create Ecore and genmodel for this data as it is described in subsection Genmodel is used as an input for the EMF to generate Java implementation of the application model. From the same genmodel, EMF.Edit is used to generate view support and a basic editor to manipulate and view the model of the application. During life cycle of the application model, model may need to be changed for many reasons. To support the model changes, the EMF provides regeneration mechanism. To avoid losing modifications, the EMF uses a tag "generated" located in generated comments for each generated class and its methods. For example, when a method is modified in generated code, the tag "generated" is replaced by "generated not". Therefore, when the EMF Codegen read "generated not", the corresponding method is not overwritten.

35

36 5 Design of SMUIML Editor 5.1. Problem Analysis Problem Solution Graphical Representation of SMUIML Visual Edition of SMUIML Presentation of SMUIML Editor Views of SMUIML Editor Actions of SMUIML Editor Monitoring Tool In this Chapter, we first analyze the problems that arise when editing a SMUIML file using a standard XML editor. Then, we present our approach to overcome most of the these problems and finally we present the SMUIML Editor and some of its functionalities Problem Analysis Figure 5.1 shows that creating SMUIML file and Java handler is made using two editors. First of all, the developer creates a SMUIML file and only then some information from this file can be used to create a Java handler. The fact that developers need two independent editors to develop multimodal applications means that all information of SMUIML files that are used by Java handler need to be done manually, being therefore more error-prone and inconvenient. Moreover, SMUIML file edition suffers from the following disadvantages: While editing SMUIML files, developers need to scroll up and down each time a new element has to be referenced. Knowing the multimodal dialog scenario that is represented by a given dialog element is possible only by redrawing an equivalent state machine. The complexity of this problem increases with the size of the file. As the adage says, a picture is worth a thousand words, we have chosen to highlight problems of editing a SMUIML file through Figure 5.2. There, arrows are used in order to show relationships between two elements. Scrolling up and down to read the SMUIML 21

37 22 Chapter 5. Design of SMUIML Editor Figure 5.1.: Editing SMUIML configuration file and Java handler cause user frustration. file is represented by showing its content in two columns, however it is to be noted that the editor would display such content in only one column. Figure 5.2.: Relationships among elements in SMUIML file Problem Solution With the purpose of make a simpler integration of HephaisTK with the client application, a GUI was designed with two goals in mind. The first goal is to provide a GUI that edits SMUIML files by means of graphical objects and generating Java handler from it. The second goal is to execute the client application directly from the same GUI. Figure 5.3 shows that the Java handler can be directly generated from a SMUIML file created with the visual editor. This is a solution to the problem of using two separate editors as higlighted above.

38 5.2. Problem Solution 23 Figure 5.3.: Java handler is generated from SMUIML configuration file which is created through GUI Graphical Representation of SMUIML As it was introduced in subsection 2.1.3, when SMUIML was described, files written in SMUIML can be subdivided into three main components, namely the input/output component, the events component, and the dialog component. In particular, we gave special attention to the dialog component in order to achieve our solution. Subfigure number 5 of Figure 5.4 presents an instance of dialog element. Dialog component represents a state machine which is a well known mathematical abstraction and has graphical representation. The decision to use this graphical representation was spontaneous (natural and unconstrained). The description of all steps involved in the process that led us to achieve this solution is as follows. Figure 5.4 offers a graphical representation for each step. Step 1 A simple machine state is drawn and a simple mapping between dialog elements and different components of the state machine is performed. The context element is mapped to state and represented by an a blue ellipse, while the transition element and attribute context of the element result are mapped to an arrow. Stage 2 As a simple arrow is not sufficient to represent the complexity of the element transition, we extend our solution presented in step 1 by adding, to each transition arrow, another shape (rectangles, labelled t1 and t2). Step 3 The content of the transition element consists mainly of two categories of elements. The first category is concerned by defining under which circumstances the transition is selected, while the second defines the result of selection of a transition. In order to define under which circumstances a transition is selected, SMUIML uses one of the the following elements: seq_and, seq_or, par_and and par_or. In order to avoid confusion through the use of a rectangular figure with a name that refers to one of such elements, this information is instead encoded by using different color and shape of the figure for each element. Step 4 To define results when a transition is selected, each action attribute of a result element is represented using a small icon with the value of attribute as a name. Similarly, a ref_trigger element is represented using a small icon with name of referenced trigger. The icon is therefore used to visually offer information about the type of the element or attribute. However, it can be noted that the procedure above does not clarify how other SMUIML sections of the file can be edited. The following subsection will address this issue.

39 24 Chapter 5. Design of SMUIML Editor Figure 5.4.: Numbers from 1 to 4 are steps of designing VPL of VPS that generate SMUIML, while 5 represents an equivalent dialog element written in SMUIML Visual Edition of SMUIML As it is described in subsection 2.1.3, when SMUIML was described, the structure of SMUIML file consists of four sections. These sections are represented by elements of the following types: recognizers, triggers, actions and dialog. The recognizer section declares input and output. Figure 5.2 shows that content of sections triggers and actions represented by XML element triggers and XML element actions are referred from XML element dialog. Recognizers During the study of SMUIML files, we realized that it is difficult for developer to know which recognizers are available in the platform. Besides, even when the developer may know which recognizers are available, writing the recognizers section within the config-

40 5.3. Presentation of SMUIML Editor 25 uration file can be tedious and difficult. In order to overcome these limitations, the generation of an XML file that contains all available recognizers from HephaisTK and uses this file to populate the section recognizers in the SMUIML file. Triggers and Actions Each trigger element used by a transition element is declared in the triggers section. Similarly, each action element used by a transition element is declared in actions section of SMUIML file. In order to have a simple visual representation of SMUIML we have chosen to represent only the dialog section as it is shown in Figure 5.4. Our solution was to decide that each time a new ref_trigger element or action attribute of element result need to be created, a popup window appears. This window acts as wizard, e.g., it helps user to select from existing trigger elements or create new one. The implementation of the graphical representation design described in this section will be discussed in the next chapter Presentation of SMUIML Editor The SMUIML Editor consists of many views. Managing these views is defined using perspective 1. A perspective determines also the visible actions and views within a window. It defines also the layout of particular views and tools that are needed for a given application. SMUIML Editor uses Java HephaisTK perspective which is shown in Figure 5.5. The remainder of this section is structured as follows: Subsection presents each view while Subsection presents accessible actions within HephaisTK perspective. Then in Subsection describes a monitoring tool Views of SMUIML Editor The workbench shown in Figure 5.5 consists of three windows that act as container of views. Project Explorer View is located on the left-hand side of the workbench and is used to manage resources of workspace as projects. Each project contains all needed files that are needed to develop multimodal application based on HephaisTK. The central area of workbench contains the SMUIML Editor. The SMUIML Editor is a multipage editor compound by two pages Dialog and Smuiml. The Dialog page uses a graphical representation described in subsection to visualize and edit visually a SMUIML file. This page uses a toolbox that consists of the following four sections. The first section presents all needed tools to create the dialog state machine and also selection tool. The Operators s section contains all needed tools that are used to define a combination of different modalities for a given transition. The Triggers s section contains tools to create trigger elements. This section is updated dynamically each time a new trigger is defined. The Actions s section contains tools to create action elements. This section is also updated dynamically each time a new action is defined. 1 A perspective is used in Eclipse as a visual container for a set of views

41 26 Chapter 5. Design of SMUIML Editor Figure 5.5.: General overview of SMUIML Editor. The Smuiml page allows to edit a SMUIML file using a tree graphical editor. This page is used generally to delete actions and triggers that are not used by the multimodal humancomputer dialog scenario. It also allows to view all elements of a SMUIML file and to create others Actions of SMUIML Editor Figure 5.6.: Partial part of SMUIML Editor toolbar. In Figure 5.6 the green rectangle highlights part of the SMUIML Editor toolbar. This part of toolbar is only visible when a Java HephaisTK perspective is open. Icon is used to create new HephaisTK project. Icon is used to create a new SMUIML file and is activated only when a project folder is selected. Icon is activated when SMUIML file is selected and is used to generate Java handler. Icon is used to change the direction of directed diagram that represents the dialog state machine. Icon is used to collapse all expended transitions and expend all collapsed transitions.

42 5.3. Presentation of SMUIML Editor Monitoring Tool SMUIML Editor provides a monitoring tool that allows to visualize in real-time the current context of the HephaisTK. This tool can be used to test a multimodal humancomputer dialog interaction with or without a multimodal client application. When this tool is used without a multimodal application it allows developer to test a multimodal human-computer dialog interaction before developing a multimodal application. This makes the SMUIML Editor acting as prototyping tool for developing multimodal applications. Figure 5.7, Figure 5.8, and Figure 5.9 are used to show how the monitoring tool is working. When the Java handler is executed the monitoring tool is launched. At this step the monitoring tool selects the state start (see Figure 5.7). When HephaisTK detects trigger that correspond to t_mouse the transition between state start and state state1 is selected. Then the monitoring tool selects the state state1 (see Figure 5.8). When another t_mouse is detected by HephaisTK the transition between state state1 and state state2 is selected then the monitoring tool selects the state state2 (see Figure 5.9). Figure 5.7.: Selection of state start by monitoring tool. Figure 5.8.: Selection of state state1 by monitoring tool. Figure 5.9.: Selection of state state2 by monitoring tool.

43

44 6 Implementation 6.1. SMUIML Editor Architecture EMF Model Specification Generation of a Basic Editor as Plug-in Model Generation Generation of a Basic Editor The EMF Generated Packages Visual Editor Page Implementation Model View Controller Integrating with the workbench Dialog Page Packages Adaptation of Generated Code In the previous Chapter we described the design of a graphical representation for SMUIML, and in this Chapter we describe its implementation. First of all, the high level architecture of SMUIML Editor is presented, followed by a description of its low level architecture. Finally, an overview of how previously generated code can be adapted to our needs is presented to conclude this Chapter SMUIML Editor Architecture SMUIML Editor was built upon the Eclipse platform, GEF and EMF. The Eclipse platform provided the core environment within which the application is executed, EMF was used to generate the basic tree editor, while GEF was used to implement the diagram editor. The architecture follows a MVC architecture and it is mainly determined by the used technologies. This architecture helps to separate components of the SMUIML Editor and therefore facilitates their modification and development. Aniszczyk [23] gives an introduction of using EMF based models in GEF based editors. Figure 6.1 shows how 29

45 30 Chapter 6. Implementation Figure 6.1.: High level architecture representation of SMUIML Editor different components are integrated to provide functionalities of the application. Figure 6.1 shows that components of SMUIML Editor can be divided into two groups. First group consists of components generated using EMF framework while the second group consists of components developed upon GEF. Both groups follow MVC architecture and share the same model EMF Model Specification In order to generate Java classes that represent a model and its basic tree editor using EMF, an Ecore model needs to be created. This model can be created from scratch or imported from other models as it was already introduced in subsection In our case, the Ecore model was created by importing the XML Schema that defines a SMUIML model. This choice was motivated by the fact that a XML Schema appears to be more appropriate, because creating an Ecore model by importing XML Schema adds metadata information to the Ecore model that describes the relationships between the Java representation of the model and its serialization. Therefore, the serialization of an instance of the Java model becomes transparent because it uses names of elements and attributes as described in the Ecore model (see Figure 6.2).

46 6.3. Generation of a Basic Editor as Plug-in 31 Figure 6.2.: Serialization of an Ecore EClass and Attribute are defined by the value of the attribute name of EAnnotation extendedmetadata. Here, Actions are serialized as actions. Figure 6.3 shows an UML obtained by Ecore diagram tool. This Ecore model is obtained from the XML Schema (see Code extract A.1) by using the EMF importing tool. This XML Schema was written as a result of the analysis of many SMUIML scripts. One change had to be made in the syntax of SMUIML in order to have a valid XML Schema that defines the SMUIML model. This change concerns the element trigger, located inside the dialog section of the SMUIML file. Such an element becomes ref_trigger instead Generation of a Basic Editor as Plug-in Once an EMF model is specified using an XML Schema and imported using the EMF tool, two models are generated: a core model, called Ecore and a generator model, called genmodel. The genmodel is a Decorator 1 of the Ecore model, providing additional information to that given in the model and needed by the EMF code generator to produce an appropriate source code. Figure 6.4 offers an overview of this process Model Generation The EMF code generator generates for each element a corresponding Java interface and method definition that represents accessors to their attributes. It also creates classes that implement these interfaces, and the factory that creates instance of elements in the model and other codes that define some proprieties like static constant reference, global variable, and so on. The generated classes are based on EMF runtime library that provides great numbers of features. For example, a notification mechanism feature allows an observer to be notified when one of its observables is modified Generation of a Basic Editor The Eclipse Modeling Framework (EMF) is extended by edit plug-in, which provides generator for a basic editor to create and edit model instance. EMF.edit plug-in generates Java classes that represent controller and viewer of MVC architecture. This editor displays the model using JFace TreeViewer and uses a basic property sheet to edit value of attributes. 1 Used to extend the functionality at run-time (see Gamma et al. [24]).

47 32 Chapter 6. Implementation Figure 6.3.: UML representation of SMUIML model generated from Ecore model using the Ecore diagram tool.

48 6.3. Generation of a Basic Editor as Plug-in 33 Figure 6.4.: From XML Schema to an EObject model. Figure 6.5 shows the architecture of the generated editor. The SWT/JFace TreeViewer class is used to display the model through an adapter object called a content provider. EditingDomain provides editing access to an EMF model (Copy, Remove, Add, etc.). Figure 6.5.: Architecture of generated editor The EMF Generated Packages The plug-in generated from the Ecore model, described in the section 6.2, has total of five packages. Here there is a short description of each package:

49 34 Chapter 6. Implementation ch.unifr.smuiml consists of Java interfaces for all classes in the Ecore model and an interface that define the factory that provides a create method for all not abstract classes. ch.unifr.smuiml.imp consists of the implementation classes for all Java interfaces defined in package ch.unifr.smuiml ch.unifr.smuiml.util consists of classes that are used to persist and load the model. ch.unifr.smuiml.provider consists of implementation classes that adapt the classes of the model for editing and displaying. ch.unifr.smuiml.presentation consists of classes that implements a functional structured editor similar to look and feel of Eclipse Ecore editor. Figure 6.6.: The EMF generated packages and their contents Visual Editor Page Implementation In order to edit the visually generated model, a GEF framework is used. Figure 6.7 shows how an EMF model is integrated by graphical editor build upon the GEF framework. This section uses a MVC architecture to describe our implementation. The dialog page is used to edit and display an existing file written in SMUIML, or create a new one visually. To implement this page editor, some changes are made into the EMF generated code in order

50 6.4. Visual Editor Page Implementation 35 to allow smuiml page and dialog page to share the same commands stacks and property sheet view, etc. These changes are detailed in Section 6.5. Figure 6.7.: Architecture of diagram page of SMUIML Editor Model The model was generated using the EMF Codegen from the EMF model described in 6.2. This model respect the four guidelines defined by Majewski (2011) [22]. According to these guidelines, the model should: store all data that may be edited or viewed by the user; provide the ways to persist the model; remain ignorant of the view or the controller; provide a way for others to listen to changes in the model; The SMUIML model does not contain x and y attributes for elements context and transitions, therefore, the model cannot record information on the placement of elements in the graphical view. The placement of items is managed dynamically using a directed graph algorithm. We have chosen this solution in order to avoid to deal with two models and its synchronization. This solution allows also the developer to focus on the semantics of the graph and leave to SMUIML Editor to place graphical elements using an automatic layout View The Graphical Modeling Framework (GEF) uses Draw2d to draw its figures. Draw2d is a lightweight system of graphical components, and unlike swing is almost purely graphical. Its purpose is only to display things on the canvas, which explains why the view does not know anything about the model. GEF ensures the communication between the model and the view by using controllers (editparts). Only one part of SMUIML model needs to be edited visually. This choice was explained in subsection 5.2.1, thus following this logic we create figure for each element in dialog section as shown by Figure 5.4.

51 36 Chapter 6. Implementation Controller The controller component consists on many editparts. Each editpart, connects an element of the model to its representation and is responsible to keep them synchronized. For example, renaming a figure in the graphical editor also changes its related element name in the model by means of its editpart. Editparts use listeners to keep the model and its graphical representation synchronized. In order to listen to SMUIML model objects, each editpart must implement the adapter interface. This interface defines an important method notifiedchanged, where changes in the model object are detected Integrating with the workbench Integration of the editor with the workbench allows the editor to use the pop-up menu, menu bar, tool bars, editors, perspective, and wizards, described as follows. Editor Actions The createactions method of the GraphicalSmuimlEditor class creates most desired editing actions, such as undo, redo, select all, delete, save, print actions, copy, paste, and so on. In order to connect standard menu items to these actions, buildactions and contribute- ToToolBar methods are implemented in the GraphicalSmuimlActionBarContributor (see ch.unifr.smuiml.presentation.graphical.actions package in source code). Persepective A perspective determines the visible actions and views within a window. It defines the layout of particular views and tools that are needed for a given application. SMUIML Editor uses Java HephaisTK perspective to provide mechanisms for task oriented resource interaction. For this project a new perspective is defined by using the perspective extension point in the plug-in registry. It contains the Project Explorer View, Property Sheet View, and an editor area. In order to be used by our plug-in, an extension point "org.eclipse.ui.perspectives" is used (see plugin.xml). This extension point defines name, class, id and icon of perspective. Attribute class of extension point refers to class that defines the initial layout for the perspective. Wizards Wizards are used to create new resources like Projects, Folders and Files in order to simplify user tasks and increase productivity. For example, in this project we used Java HephaisTK project wizard to create the project that contains all needed files, libraries to launch HephaisTK. All wizards are implemented using the org.eclipse.jface.wizard package of JFace. The "ch.unifr.smuiml.hephaistkproject.wizard" package in the code source contains all resources and Java classes used to create this wizard. Other wizards used in this project are the following:

52 6.5. Adaptation of Generated Code 37 Trigger wizard is used to select a trigger from existing triggers or create new one. Action wizard is used to select an action from existing actions or create new one. SMUIML wizard is used to create SMUIML file with name of HephaisTK project which is appended the substring "config.smuiml". The "ch.unifr.smuiml.presentation.graphical.wizards" package contains the implementation of all of these wizards Dialog Page Packages ch.unifr.smuiml.presentation.graphical consists of two classes that create an Editor with flyout palette. The flyout palette will only be visible when the palette view is not open. The editor contains a single GraphicalViewer. ch.unifr.smuiml.presentation.graphical.figures consists of classes that represents visual representation of the dialog part of the model. ch.unifr.smuiml.presentation.graphical.parts consists of classes that represent Edit- Parts that play role of controllers in MVC architecture of GEF. There is one EditPart per figure that visualized model object. Additionally, EditParts contain EditPolicies that are responsible for requesting changes to the model. ch.unifr.smuiml.presentation.graphical.policies consists of classes that represents EditPolicies, each EditPolicy focuses on a single editing task or group of related tasks. They are used inside EditPart to determine their editing capabilities. ch.unifr.smuiml.presentation.graphical.commands consists of classes that represent commands called by EditPolicies or actions. When commands are executed the model is modified. ch.unifr.smuiml.presentation.graphical.properties resolves the corresponding model element of a selected EditPart. ch.unifr.smuiml.presentation.graphical.actions consists of classes that represent actions used by a workbench part. ch.unifr.smuiml.presentation.graphical.wizards consists of classes that represent wizards used to create or select triggers or actions elements of SMUIML file Adaptation of Generated Code Steinberg et al. [8] describe two solutions to perform any adaptation in the generated model. The first solution is to change the specification of Ecore model and regenerate the Java code, while the second solution is to change directly the generated code. Depending on the kind of adjustment, one of these two solutions will be used. The following are two examples that refer to these changes that were made in this thesis: To represent a connection among elements in the GEF, model should have the connection element. Instances of a SMUIML file do not have explicit elements representing connections among contexts. Such connections are implicit and held by the attribute context of element result, contained by an element transition. To solve this problem we have chosen to modify the Ecore model by adding

53 38 Chapter 6. Implementation two elements and regenerate Java classes. Firstly a EClass, called Connection with two attributes, source and target and secondly an abstract EClass, called ConnectableNode is created and used as subclass of EClasses Transition and Context. Figure 6.8 summarizes the described modifications. SMUIML files obtained by using the basic editor generated by EMF were almost as required. Only some attributes have values encoded differently which make SMUIML files obtained by the basic editor not conform with SMUIML requirements. To solve this problem small change can be made in the resource factory implemented by Java class SmuimlResourceFactoryImpl. These changes consist of commenting lines 49 and 50, and adding NOT tag in their JavaDoc comments to prevent overwriting after code regeneration. Figure 6.8.: Modification of Ecore. 1 / * * * < copyright > 3 * </ copyright > * 5 * $Id$ * / 7 package ch. u n i f r. smuiml. u t i l ; 9 import org. e c l i p s e. emf. common. u t i l. URI ; 11 import org. e c l i p s e. emf. ecore. resource. Resource ; 13 import org. e c l i p s e. emf. ecore. resource. impl. ResourceFactoryImpl ; 15 import org. e c l i p s e. emf. ecore. xmi. XMLResource ; 17 / * * * <! begin user doc > 19 * The <b>resource Factory </ b> associated w ith the package. * <! end user doc > 21 ch. u n i f r. smuiml. u t i l. SmuimlResourceImpl 23 * / public class SmuimlResourceFactoryImpl extends ResourceFactoryImpl { 25 / * * * Creates an instance of the resource f a c t o r y. 27 * <! begin user doc > * <! end user doc > 29 * / 31 public SmuimlResourceFactoryImpl ( ) { super ( ) ; 33 } 35 / * *

54 6.5. Adaptation of Generated Code 39 * Creates an instance of the resource. 37 * <! begin user doc > * <! end user doc > 39 not * / public Resource createresource ( URI u r i ) { 43 XMLResource r e s u l t = new SmuimlResourceImpl ( u r i ) ; r e s u l t. getdefaultsaveoptions ( ). put ( XMLResource. OPTION_EXTENDED_META_DATA, Boolean.TRUE) ; 45 r e s u l t. getdefaultloadoptions ( ). put ( XMLResource. OPTION_EXTENDED_META_DATA, Boolean.TRUE) ; 47 r e s u l t. getdefaultsaveoptions ( ). put ( XMLResource. OPTION_SCHEMA_LOCATION, Boolean.TRUE) ; 49 / / r e s u l t. getdefaultloadoptions ( ). put ( XMLResource. OPTION_USE_ENCODED_ATTRIBUTE_STYLE, Boolean.TRUE) ; / / r e s u l t. getdefaultsaveoptions ( ). put ( XMLResource. OPTION_USE_ENCODED_ATTRIBUTE_STYLE, Boolean.TRUE) ; 51 r e s u l t. getdefaultloadoptions ( ). put ( XMLResource. OPTION_USE_LEXICAL_HANDLER, Boolean.TRUE) ; 53 return r e s u l t ; } 55 } / / SmuimlResourceFactoryImpl Code extract 6.1: SmuimlResourceFactoryImpl.javal: SMUIML template

55

56 7 Validation 7.1. Validation with Users Results from Involving Development Participants User Study SMUIML Editor Evaluation First Use Case Second Use Case Results and Conclusions This chapter gives an overview of the approaches used in this M.Sc. thesis to ensure that the SMUIML Editor satisfies its initial requirements. Section 7.1 summarizes results obtained by involving users in development process, and Section 7.2 shows that SMUIML Editor is functional and can create any SMUIML File, which is illustrated through two different use cases Validation with Users In this section we will present result of our work with users. The users involved in this project were categorized into two groups: Development participants This group is composed of the supervisors of this M.Sc. thesis. As the development process was incremental, their role was to ensure that requirements for each development step were progressively fulfilled. The achievement of all of the new functionalities of the application were supported by short demonstrations under which the progress was validated and new task sets were defined. One of the most important advantages of this development process was that new functionalities were added later in project as a direct result of our progress. Subsection gives more details about these functionalities. 41

57 42 Chapter 7. Validation Test participants These users were recruited from students in Computer Science at University of Fribourg. Two of them are graduate students while two others are undergraduate students. As users in this group were unfamiliar with SMUIML, the usability testing of the application was somewhat limited. Testing scenarios used by this group were solely created in order to test users interaction during manipulating graphical elements of the dialog diagram. The results were obtained by simple observation of how users interact with the application during executing the given scenario and at the end by asking them open questions. Subsection introduces this scenario followed by our observation and users feedbacks Results from Involving Development Participants Given that multimodal dialog scenarios are defined using state machine abstraction, it became clear for one of the users that some properties of state machines should be respected in order to assist users to create valid dialog scenarios. For example, when the diagram becomes large and is constituted by many states, checking if all states of dialog are reachable from starting state is not an easy task. For this reason we introduced a feature that assists developers in such check (see Figure 7.1). Figure 7.1.: Not reachable states from state start are highlighted with yellow icons. HephaisTK by convention uses the context named start as starting state. If the dialog scenario does not have state start HephaisTK picks up one of contexts as starting state which influences the dialog scenario designed by the developer. We could not test if the users are aware of this convention. To avoid situation where dialog scenario does not have a start state, every new SMUIML file created with the SMUIML Editor is created with one state labelled start. The SMUIML Editor helps to avoid most of errors by using constraint of how graphical elements can be connected to each others. Some errors can occur when developer change the name of state to an existing one. To help developer to avoid having two or more states with the same name, the SMUIML Editor highlights the second one as shown in Figure 7.2.

58 7.1. Validation with Users 43 Figure 7.2.: The state in the right shows how SMUIML highlight an error User Study To set up the user study, a small scenario was created. This scenario is used mainly to introduce test participants to the capabilities of the SMUIML Editor. We also observed how users interact with basic features of the SMUIML Editor during the execution of this scenario. This allows us to ensure that users interact with the diagram editor in the expected way. The following are the tasks that users were requested to perform: Tasks of Scenario 1. Create a diagram that consists of state start and state state1 connected with a transition. The transition goes from state start to state state1. 2. Add another state to the diagram and change its name to new_state. 3. Change the diagram to have only these transitions: from start to newstate, from newstate to state and from state1 to start. Results of User Study Our observations of users interacting with the SMUIML Editor to execute the scenario shows that users were able to execute all tasks and in different ways. For example, users can rename a state in three ways (popup menu, double click and by property viewer). The most important result arose from test participants opinions about the SMUIML Editor. Discussing with them their opinions led us to improve some features and introduce others. For example, most of users pointed out that using only an automatic layout to place graphical elements into the canvas limited their interaction. The dialog diagram in some case does not use the space of the canvas in the efficient way. Giving possibility to users to place graphical elements where they want, would have required the introduction of an undesired amount of changes. For this reason we have chosen another solution, allowing users to change the graph s direction. Figure 7.3 shows that changing the direction of diagram and zooming by collapsing all transitions allows to visualize relatively a large dialog diagram in the canvas without scrolling down-up and left-right. Last but not least, a further improvement was to allow users to change the color of graphical elements.

59 44 Chapter 7. Validation Figure 7.3.: How layout and zoom help users to visualize the graph

60 7.2. SMUIML Editor Evaluation SMUIML Editor Evaluation Dumas et al. (2010) in [2] evaluate a description language of multimodal dialog by evaluating its degree of expressiveness and usability. To asses the usability and expressiveness of the SMUIML Editor, we created two use cases, starting only from the description introduced in subsections and For each scenario we were limited only by available recognizers of HephaisTK First Use Case Description This scenario aims to create multimodal dialog that allows to control a Contact Manager Application (CMA). The CMA uses an RFID reader to read a visiting card. All visiting cards have an RFID that is used by the CMA to find its corresponding registration. If the visiting card is not yet registered by the CMA, a new form is opened and invites user to fill its fields by address and webpage that corresponds to this visiting card. For an existing contact, the CMA allows to open either a new with field to already assigned by registered address of this contact, or webpage of this contact. Multimodal Dialog Scenario Figure 7.4 shows the graphical representation of the dialog section of SMUIML file that is used for CMA and HephaisTK integration. The state start represents the context of CMA when no visiting card is read by the RFID reader. When a visiting card is placed in RFID reader, HephaisTK sends a resolvecard action to CMA and changes its context to resolvedcard. At this point, the CMA selects the corresponding contact and waits for one of these two actions: a_sendmail or a_openwebpage. To access another contact, the visiting card is replaced by a new one. Figure 7.4.: Multimodal Dialog Senario used by contact manager application.

61 46 Chapter 7. Validation Second Use Case Description This scenario aims to create multimodal dialog that allows to control an already existing application that was designed without multimodal capabilities. We have chosen for this scenario Mac OS X operation system feature called Exposé. This feature is used to quickly locate of an open window and show the desktop. To locate a window users have choice to navigate in all open windows or only windows that belong to the current active application. Multimodal Dialog Scenario Figure 7.5 shows the graphical representation of the dialog section of SMUIML file that used for the Exposé application and the integration with HephaisTK. The state start represents the context when no visiting card is read by the RFID reader. When a new RFID is placed on the RFID reader, HephaisTK check its value. Depending of this value, the a_application or the a_allwindows" action is selected and its corresponding message is sent to the client application, while changing the context to ApplicationWindows or AllWindows. Figure 7.5.: Multimodal Dialog Senario used by Exposé application Results and Conclusions By using the SMUIML Editor under the scenarios described above, we were able to develop two multimodal applications that use the HephaisTK toolkit. During our use of the SMUIML Editor we found that the degree of expressiveness of SMUIML is now separated from its degree of usability. This is for the simple reason that all features that are related to expressiveness are created by SMUIML Editor on users behalf. Using the dialog page of the SMUIML Editor allowed the graduate students involved in test participants group to follow easily our explanation of multimodal dialog scenarios shown

62 7.2. SMUIML Editor Evaluation 47 in Figure 7.4 and Figure 7.5. This confirms our hypothesis that the SMUIML Editor helps to communicate the multimodal dialog scenario in an accesible way. Last but not least, during our testing of the SMUIML Editor we did not encounter any problems creating any multimodal dialog scenario. In our experience, the dialog scenario was only limited by available recognizers of HephaisTK and our imagination.

63

64 8 Conclusions and Possible Future Work 8.1. Contributions As a whole, this M.Sc. project presents the SMUIML Editor, a graphical tool for modeling multimodal interaction. This tool provides a visual representation for HephaisTK configuration files, written in SMUIML. It provides all the functionalities required for the development of multimodal applications based on HephaisTK. To develop this application we have focused our efforts to improve usability of SMUIML as well as the provision of an integrated development tool for HephaisTK. Improving usability of SMUIML We have designed a graphical representation of HephaisTK configuration files. This graphical representation displays relationships among nodes of the graph by using state machine abstraction and therefore greatly enhancing the readability of HephaisTK configuration files by users. Additionally, we have provided many functionalities that allow to edit and view SMUIML files efficiently. These functionalities are : A zooming feature, which allows to view a large graph and also to analyze the graph according to the user point-of-interest. Controlling the graph s direction in order to achieve a more efficient use the canvas space. Several synchronized views are used in order to keep the graphical representation as simple as possible. The outline view is used to navigate in the model while property view is used to edit value of node that are not accessible from the graphical view. A dynamic tool palette allows the creation of new nodes and other related features, such as delete, undo, redo, and so on. Error detection assists users to detect states that are not reached from the starting state as well as states that have the same name. 49

65 50 Chapter 8. Conclusions and Possible Future Work Providing an integrated development environment The SMUIML Editor comes with a wizard that facilitates the creation of a HephaisTK project. Such a project consists of all the files required to run multimodal applications based on HephaisTK. The dialog diagram of SMUIML Editor is used as a debugging tool to visualize in real-time the context of a multimodal human-computer interaction, which undoubtedly would support the development of such applications Future Work In this section we would like to point out two main areas of HephaisTK project that can be explored. The first one refers to work related to SMUIML while the second one aims to set up a user study for HephaisTK using the SMUIML Editor. During our validation using scenarios presented in subsection 7.2 we have found that in order to define a low level events by SMUIML, we need to create many elements trigger. Critically, for example, in the scenario presented in Figure 7.4 we needed to describe three elements trigger to detect the low level event push down a fidget button. In order to overcome this problem, it could be worth to define a low level event in SMUIML using only one XML element trigger. Such a measure could help to reduce size of the element transition and therefore the size of the diagram. Another advantage of this approach is that it would facilitate differentiation between several triggers related to the same modality. This would therefore enhance further the readability of the diagram. Once the aforesaid improvement is performed, it could be possible to evaluate HephaisTK with users by using the application scenarios developed in this project as introduction to use SMUIML editor and HephaisTK. Afterwards, users can be asked to change the multimodal human-machine dialog scenario by using the SMUIML Editor.

66 A SMUIML XML Schema <?xml version= " 1.0 " encoding= "UTF 8"?> 2 <xsd:schema xmlns:xsd= " h t t p : / /www. w3. org /2001/XMLSchema" targetnamespace= " h t t p : / /www. u n i f r. ch / smuiml " xmlns= " h t t p : / /www. u n i f r. ch / smuiml " 4 elementformdefault= " q u a l i f i e d " > <xsd:element name= " smuiml " > 6 <xsd:complextype> <xsd:sequence maxoccurs= " 1 " minoccurs= " 1 " > 8 <xsd:element name= " i n t e g r a t i o n _ d e s c r i p t i o n " type= " i n t e g r a t i o n _ d e s c r i p t i o n " / > < / xsd:sequence> 10 < / xsd:complextype> < / xsd:element> 12 <xsd:complextype name= " i n t e g r a t i o n _ d e s c r i p t i o n " > <xsd:sequence> 14 <xsd:element name= " recognizers " type= " recognizers " / > <xsd:element name= " t r i g g e r s " type= " t r i g g e r s " / > 16 <xsd:element name= " a c t i o n s " type= " a c t i o n s " / > <xsd:element name= " d i a l o g " type= " d i a l o g " / > 18 < / xsd:sequence> < x sd:attribute name= " c l i e n t " type= " x s d : s t r i n g " use= " r e q u i r e d " / > 20 < / xsd:complextype> <xsd:complextype name= " recognizers " > 22 <xsd:sequence> <xsd:element maxoccurs= " unbounded " minoccurs= " 1 " name= " recognizer " 24 type= " recognizer " / > < / xsd:sequence> 26 < / xsd:complextype> <xsd:complextype name= " t r i g g e r s " > 28 <xsd:sequence> <xsd:element maxoccurs= " unbounded " minoccurs= " 1 " name= " t r i g g e r " 30 type= " t r i g g e r " / > < / xsd:sequence> 32 < / xsd:complextype> <xsd:complextype name= " a c t i o n s " > 34 <xsd:sequence> <xsd:element maxoccurs= " unbounded " minoccurs= " 1 " name= " a c t i o n " 36 type= " a c t i o n " / > < / xsd:sequence> 38 < / xsd:complextype> <xsd:complextype name= " d i a l o g " > 40 <xsd:sequence> <xsd:element maxoccurs= " unbounded " minoccurs= " 1 " name= " context " 42 type= " context " / > < / xsd:sequence> 44 < x sd:attribute name= " leadtime " type= " x s d : i n t " / > < / xsd:complextype> 46 <xsd:complextype name= " context " > 48 <xsd:sequence> <xsd:element maxoccurs= " unbounded " minoccurs= " 1 " name= " t r a n s i t i o n " 50 type= " t r a n s i t i o n " > 52 < / xsd:element> < / xsd:sequence> 51

67 52 Appendix A. SMUIML XML Schema 54 < x sd:attribute name= "name" type= " x s d : s t r i n g " / > < / xsd:complextype> 56 <xsd:complextype name= " t r a n s i t i o n " > 58 <xsd:sequence> <xsd:element minoccurs= " 1 " name= " content " type= " content " / > 60 <xsd:element maxoccurs= " unbounded " minoccurs= " 1 " name= " r e s u l t " type= " r e s u l t " / > 62 < / xsd:sequence> < x sd:attribute name= "name" type= " x s d : s t r i n g " / > 64 < x sd:attribute name= " leadtime " type= " x s d : i n t " / > < / xsd:complextype> 66 <xsd:complextype name= " r e s u l t " > < x sd: a ttribute name= " context " type= " xsd:anyuri " / > 68 < x sd:attribute name= " a c t i o n " type= " xsd:anyuri " / > < / xsd:complextype> 70 <xsd:complextype name= " seq_or " > <xsd:sequence minoccurs= " 0 " maxoccurs= " unbounded " > 72 <xsd:element minoccurs= " 0 " name= " content " type= " content " maxoccurs= " unbounded " / > 74 < / xsd:sequence> < x sd:attribute name= " leadtime " type= " x s d : i n t " / > 76 < / xsd:complextype> <xsd:complextype name= " content " > 78 <xsd:choice maxoccurs= " 1 " > <xsd:element minoccurs= " 0 " name= " t r i g g e r " type= " t r i g g e r _ a " 80 maxoccurs= " 1 " / > <xsd:element minoccurs= " 0 " name= " par_or " type= " par_or " / > 82 <xsd:element minoccurs= " 0 " name= " par_and " type= " par_and " maxoccurs= " 1 " / > 84 <xsd:element minoccurs= " 0 " name= " seq_and " type= " seq_and " / > <xsd:element minoccurs= " 0 " name= " seq_or " type= " seq_or " / > 86 < / xsd:choice> < / xsd:complextype> 88 <xsd:complextype name= " t r i g g e r _ a " > 90 < x sd: a ttribute name= "name" type= " xsd:anyuri " / > < / xsd:complextype> <xsd:complextype name= " seq_and " > <xsd:sequence minoccurs= " 0 " maxoccurs= " unbounded " > 96 <xsd:element minoccurs= " 0 " name= " content " type= " content " maxoccurs= " unbounded " / > 98 < / xsd:sequence> < x sd:attribute name= " leadtime " type= " x s d : i n t " / > 100 < / xsd:complextype> <xsd:complextype name= " par_or " > 102 <xsd:sequence minoccurs= " 0 " maxoccurs= " unbounded " > <xsd:element minoccurs= " 0 " name= " content " type= " content " 104 maxoccurs= " unbounded " / > < / xsd:sequence> 106 < x sd:attribute name= " leadtime " type= " x s d : i n t " / > < / xsd:complextype> 108 <xsd:complextype name= " par_and " > 110 <xsd:sequence minoccurs= " 0 " maxoccurs= " unbounded " > <xsd:element minoccurs= " 0 " name= " content " type= " content " 112 maxoccurs= " unbounded " / > < / xsd:sequence> 114 < x sd:attribute name= " leadtime " type= " x s d : i n t " / > < / xsd:complextype> 116 <xsd:complextype name= " recognizer " > 118 <xsd:sequence> <xsd:element maxoccurs= " unbounded " minoccurs= " 0 " name= " v a r i a b l e " 120 type= " v a r i a b l e " / > <xsd:element minoccurs= " 0 " name= " t r a n s l a t e _ v a l u e " type= " t r a n s l a t e _ v a l u e " / > 122 < / xsd:sequence> < x sd:attribute name= "name" type= " x s d : s t r i n g " / > 124 < x sd:attribute name= " modality " type= " x s d : s t r i n g " / > < / xsd:complextype> 126 <xsd:complextype name= " a c t i o n " >

68 53 <xsd:sequence> 128 <xsd:element name= " t a r g e t " type= " t a r g e t " / > < / xsd:sequence> 130 < x sd:attribute name= "name" type= " x s d : s t r i n g " use= " r e q u i r e d " / > < / xsd:complextype> 132 <xsd:complextype name= " t r i g g e r " > <xsd:sequence> 134 <xsd:element name= " source " type= " source " / > < / xsd:sequence> 136 < x sd:attribute name= "name" type= " x s d : s t r i n g " use= " r e q u i r e d " / > < / xsd:complextype> 138 <xsd:complextype name= " v a r i a b l e " > < x sd:attribute name= "name" type= " x s d : s t r i n g " use= " r e q u i r e d " / > 140 < x sd:attribute name= " value " type= " x s d : s t r i n g " use= " r e q u i r e d " / > < x sd:attribute name= " type " type= " x s d : s t r i n g " use= " o p t i o n a l " / > 142 < / xsd:complextype> 144 <xsd:complextype name= " t r a n s l a t e _ v a l u e " > < x sd:attribute name= " from " type= " x s d : s t r i n g " use= " r e q u i r e d " / > 146 < x sd:attribute name= " to " type= " x s d : s t r i n g " use= " r e q u i r e d " / > < / xsd:complextype> 148 <xsd:complextype name= " source " > 150 < x sd: attribute name= " modality " type= " xsd:anyuri " / > < x sd:attribute name= " value " type= " x s d : s t r i n g " use= " r e q u i r e d " / > 152 < x sd:attribute name= " c o n d i t i o n " type= " x s d : s t r i n g " / > < x sd:attribute name= " v a r i a b l e " type= " xsd:anyuri " >< / x sd:attribute> 154 < / xsd:complextype> <xsd:complextype name= " t a r g e t " > 156 < x sd:attribute name= "name" type= " xsd:anyuri " use= " r e q u i r e d " / > < x sd:attribute name= " message " type= " x s d : s t r i n g " use= " r e q u i r e d " / > 158 < / xsd:complextype> < / xsd:schema> Code extract A.1: smuiml.xsd: SMUIML template

69

70 B Setup SMUIML Editor B.1. Prerequisites Required Java Development Kit Java Development Kit (JDK) 1.5 and 1.6 are recommended for the SMUIML Editor. Required Eclipse Platform The recommended Eclipse version is Release or greater. The Eclipse Modeling Tools version package can be downloaded and installed from downloads. In this case all required eclipse features are already installed, and the system is ready for installation of the SMUIML Editor (following instructions in section B.2). Required Eclipse Features The following Eclipse features are required: Eclipse Graphical Editing Framework. Eclipse Modeling Framework Runtime and Tools. The easiest way to ensure that the dependencies between features are resolved is to use the Eclipse install wizard. The following points describe how this can be done: Select Help Install New Software (see B.1). In the field Work with of the first page of install wizard that is open, click the arrow button and select All Available Sites from the list. Type GEF in text field filter (see B.2). Select Graphical Editing Framework GEF SDK from filtered list. Type EMF in text field filter. Select EMF - Eclipse Modeling Framework SDK and click on the button Next. On the new wizard page click on the button Next. Click on the button Next, and accept the terms of agreement of each selected feature in the new wizard page. 55

71 56 Appendix B. Setup SMUIML Editor Accept all agreements and click on the button Finish. Figure B.1.: How to access the installation wizard of Eclipse. Figure B.2.: Select all available sites in first page of installation wizard. B.2. Plug-in Installation To install SMUIML Editor plug-in in Eclipse, the following steps need to be executed: 1. go to Eclipse installation folder; 2. put SMUIMLEditor.jar file located in folder plugin in the CDROM in plugin folder of your Eclipse folder installation; 3. start Eclipse (or restart it, if already running);

72 B.3. Getting Start 57 B.3. Getting Start Now Eclipse is set up for the SMUIML Editor. The first step towards using the SMUIML Editor is to insure that Java HephaisTK perspective is open (see Figure B.3). If Java HephaisTK perspective is not already open, go to Window Open Perspective Other. In the dialog window that is open select Java HephaisTK and click on the button Ok. In the case where JAva HephaisTK is already open go to File menu and choosing the New HephaisTK Project menu item (see Figure B.4). In the open wizard, fill the field name with the name you want to give to your project. Select the newly created project and choose New SMUIML File. Then, in the wizard that is open, click on the button Finish. A new file is created with name xxxconfig.smuiml where xxx is the name of the project. The newly created file is finally opened in SMUIML Editor. Figure B.3.: The red arrow shows where you can find the current open perspective. Figure B.4.: How to access to the menu items SMUIML File and HephaisTK Project.

73

74 C Common Acronyms ALM Application Lifecycle Management API Application Programming Interface CMA Contact Manager Application CVS Concurrent Versions System CWM Common Warehouse Metamodel DIVA Document, Image and Voice Analysis EMF Eclipse Modeling Framework EMOF Essential Model Object Facility EWUI Eclipse Workbench User Interface GEF Graphical Modeling Framework GUI Graphical User Interface HCI Human-Computer Interaction HephaisTK Hephais Tool Kit IDE Integrated Development Environment JDK Java Development Kit JDT Java Development Tools J2EE Java 2 Platform, Enterprise Edition JET Java Emitter Template MOF Meta-Object Facility MVC Model View Controller OMG Object Management Group PDE Plug-in Developer Environment PIM Platform Independent Model PSM Platform Specific Model 59

75 60 Appendix C. Common Acronyms RCP Rich Client Platform SMUIML Synchronized Multimodal User Interaction Modeling Language SOA Service Oriented Architecture SWT Standard Widget Toolkit UML Unified Modeling Language UI User Interface VPL Visual Programing Language VPS Visual Programing System XML Extensible Markup Language XMI XML Metadata Interchange Zest Eclipse Visualization Toolkit

76 D CD-ROM On the CD-ROM Figure D.2 of the project you will find: The source code, and compiled binaries of the project. The binaries and sources of this documentation. Various documents that were of great use during this M.Sc. thesis. Figure D.1 provides a tree view of the CD-ROM. 61

77 62 Appendix D. CD-ROM -- master_thesis //Binaries (pdf, ps, etc.) and -- appendix //sources of this document -- codes //and of the presentation. -- figures //Sources of the figures. -- chapters -- introduction -- chapter1 -- Demos //videos -- plugin //Jar files. -- source_code //plug-in s source code Figure D.1.: Tree view of the content of the CD-ROM Figure D.2.: The CD-ROM of this project