Semantic-based support for the semi-automatic construction of multimedia presentations

Transcription

1 Semantic-based support for the semi-automatic construction of multimedia presentations Kateryna Falkovych CWI P.O. Box NL-1090 GB Amsterdam The Netherlands Jana Werner CWI P.O. Box NL-1090 GB Amsterdam The Netherlands Frank Nack CWI P.O. Box NL-1090 GB Amsterdam The Netherlands ABSTRACT This paper explores the interdependencies between userdetermined interaction in the authoring process of the complex multimedia presentation and the semantic descriptions of tasks and content that allow active system support for the user during the various process phases. The domain we have chosen is the one of the fine arts. The process phases in which we are particularly interested include events such as decision making about form and function, establishing the logical structure, material collection and selection. We introduce the design of an experimental framework for semi-automatic authoring, SampLe. This exploits large mediaaware semantic spaces through semantic-sensitive authoring methods to support users mainly during the early stages of presentation design. We discuss the relations between interactionbased interface design for the system and the underlying semantic structures. Categories and Subject Descriptors H5.4 [Hypertext,Hypermedia]: Architectures, Navigation, User issues. I.7.2. [Document Preparation]: Hypertext/hypermedia, Multi/mixed media. General Terms Design, Experimentation, Human Factors. Keywords Context, semantics, ontology, interaction design, discourse structure, presentation authoring 1 INTRODUCTION The production of multimedia presentations as part of a verbal presentation or an interactive web site, is a complex, resource demanding, and distributed process with the aim of providing interesting and relevant information that suits the need of the identified audience group [10]. Most authors of multimedia presentations belong to one of two expertise extremes: either being equipped with a sound knowledge about the domain but lacking presentation skills, or being skilled in presenting information but lack a sufficient level of domain knowledge. What is needed is an approach towards semi-automatic multimedia authoring that is able to actively support any type of author in getting through the presentation making process, including events such as decision making about form and function, establishing the logical structure, material collection and selection, and presentation composition. This paper describes work in progress that explores the design of a semi-automatic framework for authoring multimedia presentations (SampLe). A presentation is understood here as a multimedia based structured discourse that allows the creator to communicate information and insights to a target audience in an interactive fashion. As more annotated information becomes available on the Semantic Web, a system such as SampLe [3] can be used to exploit large media-aware semantic spaces through semantic-sensitive authoring methods that support users in a range of purposes, such as to get acquainted with a domain, or to develop an exhaustive multimedia presentation on the topic of interest. The domain knowledge space used to illustrate SampLe covers the fine arts. The types of presentations dealt with are those typical for this domain, such as bibliography, essay, commentary, and so forth. From the system perspective, the presentation authoring process involves the manipulation of content, presentation structure and media modalities with respect to the relevant tasks required at any point during the process. To enable assistance the system has to be aware of all these components. We chose to apply Semantic Web technology to provide the system with access to domain, discourse and media descriptions about, or between, information units in the underlying semantic space. To deliver the required functionality to users we apply principles of interaction design (ID). Since exact user preferences for performing certain activities cannot be

2 predicted, the interaction design aims to provide a range of alternatives for each activity. For example, entering keywords or browsing through a set of illustrated style examples can both facilitate the search for the topic of the presentation. The opportunities the interaction design is able to offer are only possible because the underlying semantic network provides the relevant information on content and task level. The rest of the paper is structured as follows. The section about the SampLe framework introduces the core structures and describes them in relation to existing work. We then address the underlying assumptions of SampLe. As the goal of this paper is to discuss how media-based knowledge spaces can be exploited by ID-based authoring tools, following sections discuss various interaction design issues and their influence on the design of the SampLe user interface. Relevant during these discussions is always how the tasks can be supported by the underlying meta-data and mechanisms. The paper concludes with a description of future work. 2 SAMPLE FRAMEWORK The SampLe framework covers three functionalities. First it enables the creation of a presentation. A presentation in SampLe is understood as a discourse structure that describes a path through an associated semantic network, which represents the domain knowledge to be explored. Second, SampLe assists the user on various levels of expertise to access, interpret, evaluate and manipulate the complex representation structures of the domain. Finally, the system facilitates the extension of the semantic space as part of the working process of presentation authoring. It is important to notice that there is no clear-cut division between the three core functionalities and during our further discussion their interdependent nature will be clarified. Though one aim of the SampLe framework is to explore the possibilities to support all stages of the semi-automatic authoring process for multimedia presentations we focus in this paper on those phases where the author establishes the topic and related structure of the presentation and gathers the relevant information units to fill the structure. Thus, all processes with respect to final layout are ignored. 3 RELATED WORK System-supported authoring has been an active research area for a long time [7, 8]. The aim of this research direction has been tool focussed, namely by providing methodologies that facilitate the selection of appropriate interface object classes and their attributes, and placing selected interface objects in a dialog box or a menu in a meaningful, logical and consistent manner. Our approach is different, as we are interested in supporting the result of the authoring process, namely a coherent and logical presentation about a particular topic. Thus, our approach is presentation oriented. Established work closest to our own has been carried out by Bailey et al. [1] and Gibbins et al. [5]. With the work of Bailey we share the attempt to support the early stages in the design process, when the user struggles to explore and compare ideas. The approach of Bailey, however, is based on the notion of an informal interactive storyboard that addresses the needs of skilled designers to develop interactive multimedia applications first in a sketchy manner. In our work, we focus on presentations authored by users that require a more formal work environment where the authoring process leads eventually to the final presentation. More importantly, the processes in Bailey s work are not supported by explicit knowledge in the system about the process or the domain, which is our goal. The work of Gibbins et al. is of interest because mspace addresses the mapping of user-determined interaction onto a high dimensional space represented by an ontology. The aim of mspace is to explore a multidimensional space by extracting meaningful slices from it that represent those aspects of the domain the user is interested in (interface controls in the form of columns where each control is a collection of objects of the same conceptual type). We share the view that there is a need for strongly correlating the interaction behaviour of the user with the semantics inherent in the information. We see information, however, not only on an exploration level but as well as material to be manipulated (authored). Thus we try to include this sort of task semantics into the information unit while it is being manipulated with the intention of using this point-of-view on the information unit at some other time when another presentation is being built. Moreover, we have to handle a great variety of media types and are thus challenged with a greater variety of ontological descriptions. 4 AUTHORING PROCESS Getting a better understanding of the relation between information units and tasks we categorized the authoring process with respect to required tasks and related material and established four stages (see also [2], pp ), namely: Theme identification facilitates the identification of the presentation theme in terms of content and presentation settings. Content specifications comprise the topic and possible related topics. Presentation settings concern selecting media types (e.g. audio, video, text, etc.) and the desired flow of the presentation whether it be an interactive or a continuous mode. Specification of presentation structure consists of selecting and altering genre templates. A genre is a presentation structure (e.g. essay, biography) that consists of conceptual parts (e.g. introduction, description, conclusion). System-proposed conceptual parts depend on the user s choice of topic and related topics or can even be freely chosen by the user herself. The latter facilitates the user not to be limited in her creativity by too much

3 automation. Furthermore, structures and conceptual parts can be edited by the user, either by extending or reducing them, as well as changing number, size or order of conceptual parts. Collection of material allows the retrieval of information in which media items are collected depending on the current conceptual part the user is working on. Moreover, new information units can be imported or generated and links between existing information units can be altered with respect to the current authoring context (authoring task performed and role of the information unit in the current discourse). For assuring the retrieval of adequate information in the right media type, data has to be annotated with the concepts from a variety of ontologies, namely domain, discourse and media ontology. We describe these in more detail in section 5. Arrangement of material supports the specification of a coherent ordering based on rules concerning the content, discourse structure and the presentation structure the media items need to fit in. Spatial and temporal orders have to be taken into account. These four steps are iterative and can be interrelating. In stage 1 the user might have chosen, for example, a genre for the presentation (e.g. a biography) and realizes later, after having inspected the retrieved material and collected information units that a different genre (e.g. an essay) would fit the content better. She might go back, therefore, and change the genre. To enable such wide-ranging support the system requires knowledge not only about the essential structural elements of the data and their attributes that are required to establish the contextual aspects of an information unit in a given domain but also the conditions in which the transformation from information into knowledge takes place, and the way users utilize these processes. 5 ANNOTATIONS IN SAMPLE The framework s intelligence lies in the ability to understand user intentions through the task performed during the authoring process. The mapping of the authoring context on the various meta-data structures provides the potential for appropriate assistance. This mapping between task and internal system content representations is supplied by the meta-data update process. Enriching existing annotations will help the system to acquire more knowledge about the elements and their roles in the different external contexts. Thus, system suggestions will become more flexible and helpful for users. Consider the following examples. Domain annotations contain meta-data that describe, in our case, the domain of fine arts (for example descriptions provided by the Rijksmuseum in Amsterdam). The users of SampLe, however, might find that other meta-data is also relevant for a specific information unit that would give a better description of the unit s content. If the user adds completely new information the relevant content description schemata need to be instantiated (using Getty s vocabulary, e.g. the Art and Architecture Thesaurus [4]). The information units must then be included into the semantic space by establishing the relevant discourse structures, using the discourse ontology as support. Discourse annotations are designed to establish the communication role an information unit can play. Thus, each information unit represents the point of view with respect to discourse. In such a way it is possible that a user can enrich the existing annotation of an information unit by using the same information unit in a different discourse context (an information item initially annotated as introductory is used within the middle section as an explanation). The system would update the discourse annotations of this information unit with the reference to the current phase and task of the authoring process. User-related annotations are at the beginning a simple representation of stereotypical user descriptions that allow the assignment of the user to a particular group (art/not art student, teacher, art expert, etc.), where the group specifies basic preferences and expertise levels with respect to content knowledge and presentation skills. This model is important as it determines the mapping between the intention of the task and related material. Each new interaction with SampLe would enrich user-related annotations with the result that each user leaves traces in the semantic space. In each of the cases described above, updated or newly created annotations will be accompanied by the annotations representing the current authoring step. In this way the manner of using information in a certain process step is notified. This leads to an evolving enrichment of the systems meta-data framework and hence facilitates the system s ability to represent particular information from different identified viewpoints. The more users deal with the system, the more information is gathered and the more flexible and accurate assistance can be offered by the system.

4 Figure 1: SampLe interface example (see 6 TOP LEVEL DESIGN DECISIONS When designing the framework, underlying structures and mechanisms interaction design plays an important informative role. We understand interaction design as designing interactive products to support people in their everyday and working lives. In particular, it is about creating user experiences that enhance and extend the way people work, communicate and interact. ([9] p.6). In this section we describe the application of the introduced concepts and ideas into an interface mock-up, as presented in Figure 1. A detailed description of the interface and design decisions is provided in [11]. The mock-up helped us to relate the various authoring tasks with the required content and thus model the necessary meta-data structures. The latter we are currently implementing, including the related exploration and maintenance methods. Thus, whenever we state system in the following discussion we refer to the mock-up, as the state of implementation cannot yet be referred to as a working system. The aim of this interface is to visualize the various tasks and related actions for the presentation authoring process in the domain of visual arts to be used by a diverse user group covering all ranges of domain expertise and presentation making skills. The main aspect of the design, however, was to identify the alternative ways to support the various stages of the authoring process by using the available knowledge of the associated semantic space. The modelling of these alternatives provided us with the insights what mapping between task and semantic support through the system in a practical case actually meant. Based on the 4-stage model of presentation authoring (see section 4) the interface should only represent characteristics of the tasks as it cannot be expected that the user has an idea about the underlying representations of the content semantics. A second decision that influenced the interface design was that the two authoring phases collection of material and arrangement of material were combined into one workspace as the overlap of sub-tasks in both tasks turned out to be too substantial. The grouping of tasks and related actions for the various authoring phases resulted in an important outcome, namely that SampLe does not force a sequential work order on the user but suggests a natural order. This means that the user is free to approach the authoring process from every possible angle and SampLe would try to support this. It must be pointed out, however, that the system would need some basic information from the user to be able to be helpful. For example, without information on which sort of presentation the user would like to build or what the topic of the presentation supposed to be about, the system can be helpful only in the exploration task. Thus, it is wise to approach some basic tasks first but the user is still free to choose the level of detail. As a result SampLe can support authoring scenarios, such as authoring under tight time constraints or authoring based on leisure-based knowledge space exploration.

5 The third design decision was that SampLe should provide several ways of approaching a solution for task resolution. Users could thus be provided with the greatest possible freedom by applying a work-style that fits their habits. The resulting overall structure of the SampLe interface thus consists of a mix of traditional window workspaces, a web environment and a 2D+ information space as outlined in Figure 1. The top-level navigation is composed of three axes that span a 2.5D-space. Each of the three axes takes up a different dimension of information. The horizontal axis supplies the three stages that have to be traversed for building a presentation. It is important to note that the three steps fulfil two functions. Not only do they provide the user with navigation and orientation functionality, such as listing the relevant tasks associated with a particular phase. They are also used by the system to determine with which phase the user is currently active and with which task within that phase. This information is important to facilitate the mapping of external actions with the internal concepts if required. The lateral axis takes up general functionality, such as the help and tutorial functionality that is reachable throughout the whole authoring process. The vertical axis carries keyword links about all the decisions the user has taken within the presentation building process, such as the name of the chosen topic or genre, or presentation settings, such as duration or size of the presentation. This summarized information provides a quick overview and can also be used to navigate between decisions. All three navigation-axes literally lead to the point of origin, where the symbol of the final presentation is positioned, which graphically implies that using the functions and links from all axes leads the user to her goal of building a presentation. The dominant work area is reserved for the windows that provide the workspaces for the user s current task. An important high-level design decision was that each window is equipped with information functions explaining to the user what the current task is and what related actions can achieve. The windows are arranged positively in a reasonable contrast with the blue-grey background. Thereby they appear to be in front of the other elements and are perceived as being more important. Graphical elements such as lines, boxes, fonts of different shades of white and the blue-grey background are used to group or separate information. For the user the screen is now intuitively divided into areas of high importance and areas of secondary information, which will be found if specifically looked for. The following sections highlight tasks and related actions within the three phases that are supported by the associated semantic space. Though the interface design generally facilitates various approaches to structure a presentation, our example describes an economic way of presentation building, namely to initially identify a presentation subject, followed by choosing a genre and then finally filling the established structures with the adequate content. We now describe a number of tasks that illustrate the use of structures required to achieve such services as well as those that demonstrate how actions can be used to alter the semantic space. 7 THEME IDENTIFICATION The first decision to be made within the chosen authoring process is the choice of a presentation topic. At this stage a number of options would be offered by the system depending on how well a user knows what she wants to specify. Pre-determined search If a user has a clear idea of her topic an input field is provided, where a term can be entered. The system then would verify whether the entered term matches one of the concepts in the domain ontology. If this is the case, the second step is to check whether there is correspondent data in the system repository that is annotated with this term. All the elements that have matching content will be retrieved and presented to a user in one of the two available forms: in a list or in a map-view. Both of these forms depend on the meta-data structure. The list of search results shows retrieved documents arranged in alphabetical order. Each item in the list contains a short description of the corresponding element and an image, if applicable. For example, the search for De Stijl will return elements related to the movement such as De Stijl art works, artists, De Stijl magazine and so forth. An item about an artist will contain an image (photo or portrait) together with some introductory text. In order to retrieve the text the system would make use of the discourse annotations attached to the elements. If more detailed information about the particular element is required, a user can click on that element to get more exhaustive description. In this case the system will retrieve documents that are instantiated with the discourse role the concept description. Thus, already at the first stage of the authoring process knowledge about domain, discourse and media types of the elements get involved into the process of user assistance. Figure 2 presents the map-view. It shows elements distinguishable by the media type. Links between the elements represent conceptual connections between them.

6 According to the user preferences, short descriptions of each document can be added to the view together with thumbnails for images. Selection of one of the elements from the map results in more detailed description of this document to be shown at the bottom of the screen. The same mechanism for getting even more detailed information is applied, as the one described in the list view. Partially determined or undetermined search In the situation where a user has a rough idea about what she is looking for a search restricted to categories together with browsing can be applied. The categories are provided in a list and are derived from the top-level domain concepts (classes in the domain ontology) such as Artist, Artefact, Movement etc. If a user is not familiar with the subject or wants to know more about the term presented in the category list, she can explore the knowledge space by selecting a category and getting a set of sub-categories (by choosing Artist in the main category list a user will be provided with the list of all artists available). If a user decides to get more information about the particular artist, she will get some introductory facts about this artist containing a number of links to the related documents. A user can then explore the knowledge space by following the links. These links are derived from the relations between the information units in the semantic space (note, the system would try to present all available media types). As a user continues the search in one direction (looking for the information about a particular artist and his works), more detailed documents will be represented to a user. See the window in the bottom right corner of Figure 1 as an example. This is an effective way to explore the structures of a knowledge space of which the user might not be aware. Moreover, it is also a convenient way to classify a user s Figure 2. The map-view own knowledge and understanding of how the imagined topic fits into the material contained inside the system. This technique also provides ideas for new relations to investigate and connect in a presentation. In the unlikely case that the user does not know at all what topic to choose for the presentation, the system would still be able to provide support by searching through existing presentations stored in the repository (as mentioned earlier, a presentation is understood in SampLe as the path with a point-of-view through the semantic space presented in a particular style). Presenting the most popular or those built by users of the same knowledge level might lead to new sources of inspiration for the user. Once a topic is found, the system would show how much, and what, material is available. Thereby, the user can estimate, whether the potential is going to fulfil her demands in terms of variety and depth. Search for related topics/characters. The user is lead to optionally choosing related topics in a similar way as described for the topic search. Apart from those search options, we adopted a new way of content search into SampLe to provide yet another feature of creativity support, namely creative browsing (see the main window in Figure 1 as an example note the collage is animated). Here the system would show information units in a directly perceptible way and associates them towards each other. The presentation mode changes from an active choice mode into a passive mode that is breaking the workand thought-pattern of the user. The user watches the stream of related information units, all related to the already chosen topic, such as possible information in form of images or headlines of documents, videos or even audio files. In this stream the user can now pick information that seems relevant as sub-topic of the presentation. Depending

7 on the choices made by the user the system would adjust the ongoing information stream according to the domain metadata. Note, the user can go backwards and forwards in the stream as wished as the system would keep the spatial and temporal order in memory. After reviewing the chosen topic and related topics and the available material, general presentation settings, mostly taken from the SMIL standard, can optionally be specified, such as length or bandwidth. That information supports the adequate presentation of content later on. 8 SPECIFICATION OF THE PRESENTATION STRUCTURE This stage leads a user to the choice of presentation genres such as essay, description, biography, etc. The order in which these genres are presented is ranked in relevance, depending on the topic (main character) and the related topic (related character). For example, if a user has chosen an art movement as the main character, the essay genre will be first in a list followed by, comparison, biography and so on. The user chooses one of these discourse structures, e.g. essay and then gets suggestions for detailed essay structures, where each structure supplies concrete headlines that hint at the contents to be potentially presented in this section. The user can choose one of the proposed genres and move on to the next phase. Alternatively, a user can build an own presentation structure or alter one of the proposed ones. This functionality is supported by the underlying knowledge the system would have about the discourse structures and rules for their adjustment. The system contains template presentation structures for each genre, distinguishable by the main character (a concept from the domain ontology, e.g. Movement). Having the concrete instance of the main character ( De Stijl ) the template structure can be adjusted based on domain knowledge and narrative rules. Having the concrete instance of the related character the structure can be adjusted even further. If a user performs alterations to a proposed template structure, the system would store the new altered structure in the repository with the related specification of the external context in which this structure was build. Moreover, altering the structure may reveal new rules for the composition of a particular discourse structure. Thus, the mechanism of altering the template structures enriches system knowledge and therefore its flexibility in future. 9 COLLECTION AND ARRANGEMENT OF THE MATERIAL After the user has finished working on the structure of her presentation, she is lead into the content section. The created presentation structure is represented as a clickable list of links, each of which represents a particular section of the presentation. By clicking on the section name elements with the correspondent content ( De Stijl ) and correspondent discourse structure (definition) will be retrieved and shown either in a list or in a map view as described earlier (see Figure 2). The given information units and their relations are just a first proposal and can be altered, rearranged, modified, complemented etc. by the user. Additional personalized views, in terms of semantics or structure can be offered, like viewing only edited documents, certain media types, etc. The user can get a detailed view on the information presented in the map or list by clicking on them. The detailed view will open in a separate window, so that several items can be worked on at the same time, for example to compare or establish links between them or between sections of their content. In case, the user is not happy at all with the provided content, she can go back and alter the presentation structure or change her preferences about main or related characters. The retrieval of relevant elements for each section is possible due to the knowledge the system have about their content (domain annotations) and their discourse structure (discourse annotations). After a user has selected an information unit or a larger set of information units to be used in each section of the presentation, those elements could be arranged automatically by the system, based on the domain relationships. Alternatively, a user can establish the order she finds appropriate. As pointed out earlier, the facilities described so far only cover the stages of presentation planning and material gathering. The final layout decisions are not covered here. 10 CONLUSIONS The framework for a semi-automated and knowledgesupported authoring environment for multimedia presentations is only at the beginning at this stage. We have identified the main components of the authoring process and defined the related representation structures that allow a syntactic and semantic support of the user during the various authoring phases by the SampLe. There are still a number of challenges, though, that need to be addressed. The most pressing at the moment is to finetune the user interface based on the cognitive walkthrough method described in [6]. It is important that we are sure that the described tasks and related visual representations fit user requirements as they form the basis for the mapping for the system s representations of the content and point-ofviews on it in the semantic space. Moreover, we have to determine how structures can be originated and then altered depending on mere user action monitoring. We are aware that the described approach is to some extent idealized as we argue that users can alter any aspect of an existing semantic space. This is not the case for knowledge spaces as provided by the Rijksmuseum. Thus, questions of ranking user information and alterations are of importance and need to be investigated. At the moment we assume that there will be two associated semantic spaces, one is the

8 original and the other contains the alterations of the users. The crucial part here is the synchronisation of the two. The presented ideas should be understood as the first step in the direction of an epistemology-aware knowledge representation where a system utilizes a multi-semanticlayered knowledge space to support the user on a syntactic and semantic level with respect to content and skills as required. Thus we advocate authoring where each participant in a process is doing what it can perform best the user provides creativity and the machine is aware of it and provides to make it happen. 11 ACKNOWLEDGMENTS The work described in this paper was funded by the Dutch national NWO ToKeN2000 CHIME project. The authors also wish to thank our colleagues Lynda Hardman, Jacco van Ossenbruggen, and Lloyd Rutledge for useful discussions during the development of this work. 12 REFERENCES 1. Bailey, B. P., Konstan, J.A., & Carlis, J.V. (2001). DEMAINS: Designing Multimedia Applications with Interactive Storyboards. In Proceedings of the 9th ACM International Conference on Multimedia, pp , Ottawa, Canada, Sept Oct. 5, Barfield, J. Design for New Media Interaction Design for Multimedia and the Web. Pearson Education Limited Falkovych, K., Nack, F., van Ossenbruggen, J., and Rutledge, L. SampLe: Towards a Framework for System-supported Multimedia Authoring, CWI technical report INS-E0302, August Getty Vocabularies (2004): AAT: abularies/aat/ ULAN: abularies/ulan/ TGN: abularies/tgn/ 5. Gibbins, N., Harris, S. and schraefel, m.c. Applying mspace Interfaces to the Semantic Web mspace-model.pdf 6. John, B.E. and Mashyna, M. M. Evaluating a multimedia authoring tool with cognitive walkthrough and think-aloud user studies. Carnegie-Mellon University. Department of Computer Science; 1995/ Kim, W. C., and Foley, J. D. DON: user interface presentation design assistant. Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technology, pp , Snowbird, Utah, United States, Model-View-Controller Pattern 9. Preece, J. Interaction design: beyond human-computer interaction. John Wiley & Sons, Tufte, E. R. Envisioning information. Graphic Press, Connecticut, Werner, J. Investigation of Methods for User adapted visualization of Information in a Hypermedia Generation System. Master Thesis, University of Applied Sciences Darmstadt, Germany, 23 January 2004.