Semantically Enhanced Hypermedia: A First Step

Transcription

1 Semantically Enhanced Hypermedia: A First Step I. Alfaro, M. Zancanaro, A. Cappelletti, M. Nardon, A. Guerzoni ITC-irst Via Sommarive 18, Povo TN 38050, Italy {alfaro, zancana, cappelle, nardon, annaguer}@itc.it Abstract The paper introduces a framework to automatically build hypermedia links from a semantically annotated repository of multimedia data. The system architecture is based on a relational database accessible through XML queries on an HTTP connection. A shallow semantic representation is encoded as a set of key words that correspond to entities in the domain. This representation is used to annotate the texts and the images in the database. Communicative strategies are then employed by the graphical interface to dynamically produce links to the data. 1 Introduction Typically, hypermedia systems are manually prepared by content experts and technical editors. The preparation phase consists of besides preparing the content material deciding how different pieces of information should be linked together. These links are then explicitly inserted in the texts or multimedia content. One of the main drawbacks with this approach is that as the hypermedia grows, it becomes increasingly difficult to add new material as this needs to be manually lin ked to that already existing in the system. In this paper, we describe a framework for creating a multimedia database semantically annotated that can be used to automatically build hypermedia links. The main disadvantage of our approach is the need for e xplicitly annotated data with respect to a semantic model. The research trend on semantic web (Berners-Lee et al., 2001), however, has the potential to provide a suitable infrastructure to create these kinds of repositories. In the following section, we will describe the system architecture and we will briefly discuss the semantic model employed together with the annotation format used. In section 3, we will present a walk-through of a hypermedia developed on a museum setting. Section 4 will discuss the set of semantic strategies used for automatically building hypermedia from a semantic multimedia database. 2 Semantically Annotated Multimedia Database The Multimedia Database (MMDB) allows the storing of a repository of multimedia information (i.e. text, images, speech, audio, etc.) and its annotation with respect to a shallow representation of its meaning. It allows the retrieval of data chunks with respect to topics (e.g. all the texts with a given topic) and to make some limited semantic inferences on topics (e.g. all the texts whose topic belongs to a given class). The MMDB is intended as a module in a system to build adaptive presentations of information; it is both a data repository and a shallow knowledge base system. 2.1 Topic Classification The knowledge about the domain, in our case life in the Middle Ages, is encoded as a set of key words called topics representing entities, such as characters and animals, and processes, such

2 as hunting and leisure activities. At this phase of the work, only one relation can be defined between topics; that is, the member-of relation that denotes that a given topic belongs to a given class. For example, the topic fox_hunting is in a member-of relation with the topic hunting, which means that fox_hunting is a form of hunting. 2.2 Text Annotation Since the system was designed to navigate through paragraphs rather than entire documents, paragraphs within each document were explicitly marked. A title was assigned to each paragraph in order to use it as a text anchor for the links to be displayed on the interface. The meaning of each paragraph was then attributed assuming that it discussed a single concept already represented by a given topic in the knowledge base. This semantic annotation, though simple, allows the classification of text according to what is being discussed. For example, it is possible to search the database for all the paragraphs that discuss, in one way or another, hunting. This is possible simply by looking for paragraphs for which (a) the main concept is the topic hunting or (b) the main concept is one of the topics that are in a member-of relation with hunting. A second type of semantic annotation attributed to each paragraph assumes that aside the single concept, a paragraph also introduces other entities that, although not the central meaning of the text, can nevertheless be used to provide smooth shifts toward new arguments. These entities, still represented as topics in the knowledge base, are called the paragraph s Potent ial Focus List (PFL). Figure 1 illustrates the semantic annotation for one of the texts used in the hypermedia introduced in section 3 below. <paragraph title="fox Hunting" concept="fox_hunting" pfl="fox,badger"> On the right there are two hunters coming through the snow, each with two dogs on a lead. One of the hunter's dogs is making for a bush in which a badger is hiding. The whiteness of the snow-covered countryside is broken by the low bushes and the fir trees under which there are some foxes. </paragraph> Figure 1. Semantic annotation of paragraph 2.3 Image Annotation As with the document-paragraph situation, the semantically relevant element was often not the image itself but the scenes it contained. The images, thus, are sub-divided into details that could be exp ressed in a single topic, such as a hunting or a harvesting scene. A detail is represented by its bounding box; that is, the region occupied by the scene in the image coordinates. Depending on the complexity of the information, the details can be embedded inside other details. 2.4 System Architecture The back-end of the system is organized as a relational database (implemented in mysql). In order to provide a way to connect the database to any type of graphical user interface, a query engine has been developed to retrieve data from the MMDB through a standard HTTP connection. The engine, implemented using Java Servlets, transforms the request in a list of SQL queries and returns the data in XML format (figure 2). The MMDB is able not only to provide the actual data but also to perform semantic queries, for example, retrieving all the texts that have a concept that belongs to a certain class, thus making the implementation of the semantic strategies of the interface (section 4) easier.

3 Figure 2: System Architecture 3 The Torre Aquila Hypermedia: A Walk-through Our system has been conceived as a universal tool that can be applied to any multimedia domain. As a case study, we have chosen to work with documentation about the Cycle of the Months of Torre Aquila at the Buonconsiglio Castle in the city of Trento, Italy. This fresco is composed of eleven panels (each one representing a month) painted during the 1400s and illustrates the activities of aristocrats and peasants throughout the year. The fresco introduces a number of characters as well as many different activities, from falconry to wine harvesting. Figure 3: Hunting detail in January Figure 4: General presentation about hunting Figures 3 and 4 represent a snapshot of an interaction with our system. In figure 3, an image of a detail from the month of January is seen along with the text (lower left box) that describes the two hunters chasing a fox and a badger. In the two boxes on the top left side of the screen, the system generates links to related information. Given that the text describing the hunting scene explicitly mentions foxes and badgers, informative texts about foxes and badgers in the Middle Ages are generated as link proposals. Similarly, as the image represents a hunting beat, a general text about hunting as an activity is also suggested (i.e. new link generated inside one of the two top boxes). Figure 4 illustrates the interface that would appear when the user clicks on this last suggestion. On the right side of the interface, the detail of the month of January has disappeared and the new text is displayed in the box on the bottom left, while on the top left boxes, new navigating options

4 have been automatically generated. These are links to details about hunting beats repres ented throughout in the Cycle of the Months; namely, Fox and Badger in January, Falconry in July, August and September, and Bear Hunting in November. This example demonstrates three different communicative strategies. The first one (links to informative texts about foxes and badgers in the Middle Ages, Figure 3) employs information about potential foci in the current text in order to propose links to other texts that elaborate one or more of these foci. The second strategy (a general text describing hunting, Figure 3) uses the idea that the hunting beat depicted in the month of January is a particular example of hunting as an activity and thus proposes a link to another text that describes this activity in general terms. The third strategy (Figure 4) recognizes that when the current text describes a general activity or idea, it would be appropriate to propose links to specific examples that may be found in the database about the current topic. By using the three communicative strategies just described, the system is thus able to propose new information to the user following a semantic approach. In other words, the system navigates the information space and makes consequent decisions about what new information could be interesting to the user by utili zing the knowledge of what the user is currently exploring on the interface. 4 Implementation of the Strategies A communicative strategy is a mechanism to build links between relevant information. Each strategy is composed of two steps. First, a rule that defines how to find a new topic given the current topic, and second, a query in the database to retrieve all the information, i.e. paragraphs and image details, indexed by the new topic. For example, the three strategies introduced in Section 1 can be defined as follows: a) Follow the PFL Strategy: This strategy chooses the new topic among those of the potential focus list of the text currently being displayed on the interface. The database is then queried to retrieve all the paragraphs that have been annotated with that topic as the main concept. For each paragraph found, a new link is created on the interface in one of the two top boxes seen in figures 3 and 4. b) Generalization Strategy: This strategy chooses the new topic as the class to which the main concept of the text currently being displayed on the interface belongs. As with the above strategy, the database is then queried to retrieve all the paragraphs that have been annotated with that topic as the main concept and for each paragraph a new link is created. c) Class members Strategy: This strategy is applied when the main concept of the text currently being displayed on the interface is a class. In this case, the new topic is chosen among the members of that class. The database is then queried to retrieve all the paragraphs that have been annotated with that topic as the main concept. For each paragraph found, a new link is created on the interface. When a link is clicked on the interface, the corresponding paragraph is displayed and the topic that represents the main concept is used to query the database to search for a new image detail to propose. If that detail contains other embedded details the picture is transformed into an image map where details become clickable areas. If one of the details is clicked, the corresponding topic is used to search the database for a new paragraph (and possibly a new detail) to propose to the user, thus the three rules above are re-applied to build new links to other relevant information.

5 5 Conclusion In this paper, an architecture to dynamically link multimedia content previously annotated with respect to a shallow semantic representation has been presented. The advantage over traditional hypermedia lies primarily in the ease with which the system can be updated since new data can be automatically connected to the rest of the information. Similar works are being conducted by other research groups. In particular, the Multimedia and Human-Computer Interaction group at CWI in Amsterdam is working on a tool to automatically infer semantic relations between media objects and then map them onto spatial and temporal relationships within SMILE files (Little et al., 2002). They, too, have an agenda to exploit semantic web technology for hypermedia in the automation of some tasks related to hypermedia creation and management (van Ossenbruggen, 2002). While in both theirs and our system semantic model are a crucial part of the strategies used for the automatic generation of the hypermedia, CWI focuses on the generation of the interface layout for the hypermedia, rather than on link construction; as is the case with our system. The research field of adaptive hypermedia is also relevant to our approach in that it studies the use of semantic relations for tailoring the hypermedia to a specific user; in particular, works by (Kobsa et al., 2001) and (Brusilovsky, 2001) have investigated many relevant issues. By the same token, our previous work done on Macronodes (Not and Zancanaro, 2000) provided an important stepping stone to our research in that it analyzes the structure and annotation of texts for a multimedia presentation. 6 Acknowledgments This work has been conducted in the context of the PEACH project (Personal Experience with Active Cultural Heritage, Stock and Zancanaro, 2002) funded by the Autonomous Province of Trento. More information on PEACH can be found at References Berners-Lee, T., Hendler, J., & Lassila, O. (2001). The Semantic Web. Scientific American, May. Brusilovsky, P. (2001). Adaptive hypermedia. User Modeling and User Adapted Interaction, 11(1/2), Kobsa, A., Koenemann, J., & Pohl, W. (2001). Personalized Hypermedia Presentation Techniques for Improving Online Customer Relationships. The Knowledge Engineering Review 16(2), Little, S., Geurts, J., & Hunter, J. (2002). Dynamic Generation of Intelligent Multimedia Presentations through Semantic Inferencing, In Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, September. Not, E., & Zancanaro, M. (2000). The MacroNode Approach: mediating between adaptive and dynamic hypermedia. In Proceedings of International Conference on Adaptive Hypermedia and Adaptive Web-based Systems, Trento, Itay, August. Stock O., & Zancanaro M. (2002). Intelligent Interactive Information Presentation for Cultural Tourism. Invited talk. In Proceedings of the International Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems. Copenhagen, June. van Ossenbruggen, J., Hardman, L. & Rutledge, L. (2002). Hypermedia and the Semantic Web: A Research Agenda. Journal of Digital Information 3(1), August.