The Ontology for a Multi-Sensory Media Service

Transcription

1 The Ontology for a Multi-Sensory Media Service Seungwon Oh and Minsoo Hahn Digital Media Lab, Information and Communications University, Seoul Korea {aegis901, mshahn}@icu.ac.kr Abstract- Under the development of ubiquitous technologies, people need a novel media system. That means the multi-sensory media service which satisfies the five senses and makes people feel realistic. It is necessary to build knowledge about sensory effects and devices in order to provide the service. In this paper, we propose an ontology-based knowledge modeling. The proposed ontology model consists of Ontology and Device Ontology. The Ontology represents semantic information about multimedia contents and sensory effects synchronized with the contents. The Device Ontology represents semantic information about devices generating sensory effects. By using the model, we can infer sensory effects and their attributes from the predefined semantic information of multimedia contents. We proved the usefulness of our ontology model by showing results according to scenario-based experiments. Keywords - ontology, sensory, media, reality, semantic, representation. 1. Introduction Ubiquitous computing has become an integral concept in a variety ofareas. Through ubiquitous technologies, the devices for daily use have had various functions and features over traditional functions and features. In other words, a traditional environment which had just predefined simple functions has been changed to an environment providing user-focused complex services with ubiquitous technologies. The remarkable development of ubiquitous technologies indicates the necessity ofa rather improved novel media system. Now, people can enjoy an audio-visual multimedia content at home equipped with advanced audio-visual devices. However, it entertains only our eyes and ears. In the future, people will need the novel media service which satisfies the five senses and which lets people feel realistic [1]. In this paper, we will use the 'Multi-Sensory Media Service' as the terminology of this noble service. Figure 1 shows the architecture of the multi-sensory media service. I III As shown in figure 1, the service provides users with sensory effects generated by sensory devices and audio-visual contents. A media player not only plays contents but also communicates with other sensorydevices. In this case, a sensory effectmeans an effect to augment feelings by stimulating human sensory organs according to a multimedia content. A sensory device means a device to generate sensory effects such as light, wind, heat, incense, and etc. In order to provide the multi-sensory media service, many related researches should be studied. First, noble contents have to be created and devices which can make sensory effects have to be developed. Next, the service requires information about sensory effects for representing multimedia contents and sensory devices for generating sensory effects. Finally, the service needs a mechanism which manages the information and controls sensory devices. In order to meet the requirements for the multi-sensory media service, we propose an ontology-based semantic knowledge representation for the service. Ontologies are useful in building machine-readable semantic knowledge. Ontology-based semantic knowledge has important advantages. First, it is easy to share knowledge between heterogeneous devices. Next, it has a good structure to expand knowledge with increasing contents and devices. Last, it is useful for knowledge reuse and logical inference based on a semantic structure. Therefore, we proposed an ontology-based knowledge modeling. The proposed ontology model represents semantic information about multimedia contents, sensory effects synchronized with the contents and sensory devices. 2. Related Work Ubiquitous computing is a computing paradigm that a computing power becomes invisible in our lives. We can easily access to computing power anywhere, anytime and anyway through intelligent interfaces. Mark weiser says about Ubiquitous computing as follows: "Its highest ideal is to make a computer so embedded, so fitting, so natural, that we use it without even thinking about it." [2]. One ofthe main goals of ubiquitous computing is to support relevant information when it is needed. Therefore, in order to provide a service in ubiquitous environment, a knowledge modeling which represents the context about a user or environment is necessary. Ontologies are widely used for modeling context in ubiquitous environment because an ontology is useful in knowledge sharing, logic inference and knowledge reuse [4]. There are many researches that utilize ontology to provide context-aware services, for examples, CoBrA-ONT (2003) [3], ISBN Feb ,2009 ICACT 2009

2 CONON (2004) [4], SOUPA (2004) [5], GAS (2005) [6], and Ubiquitous User Modeling [7]. First, CoBrA-ONT (Context Broker Architecture-Ontology) represents the infonnation about places, agents, events and their associated properties in an intelligent meeting room domain [3]. In order to support pervasive context-aware systems, CoBrA-ONT was developed as a component of the Context Broker Architecture. Second, CONON (Context Ontology) is an ontology which has a hierarchical structure. In pervasive computing environments, CONON defines context modeling and supports logic-based context reasoning [4]. CONON consists of an upper ontology and a domain-specific ontology. The upper ontology represents general concepts about domain-independent context and the domain-specific ontology represents domain-dependent concepts for extensibility. Third, SOUPA (Standard Ontology for Ubiquitous and Pervasive Applications) was developed in order to model and support pervasive computing applications [5]. SOUPA was implemented with OWL. In order to represent intelligent agents, SOUPA includes various concepts such as user profiles, actions, events, space, time, beliefs, desires, intentions, and policies for security and privacy. In order to represent the above concepts, SOUPA references other ontologies, for example, the COBRA-ONT, the Regional Connection Calculus, the Friend-Of-A-Friend ontology, the spatial ontologies in OpenCyc, the DAML_, the MoGATU BDI ontology, and the Rei policy ontology. Fourth, GAS (Gadgetware Architectural Style) Ontology is an ontology that was developed in order to represent the semantics of the main concepts of a ubiquitous computing environment and define their inter-relations [6]. The GAS Ontology consists of the GAS Core Ontology and the GAS Higher Ontology. The GAS Core Ontology describes the common language to communicate and/or collaborate between different domains. The GAS Higher Ontology represents both the description ofeach domain and its acquired knowledge. Last, Heckmann proposed a complete framework to realize the new ubiquitous user modeling [7]. SITUATIONALSTATEMENTS and USERML were developed on the syntactic level, while UBISWORLD ontology and GUMO, the general user model ontology were developed on the semantic level. The SITUATIONALSTATEMENTS represents descriptions of situations like user model entries, context infonnation or low-level sensor data. Previous ontology researches focused on the user's state and behavior for context-aware services. Many useful ontology models were implemented for context-aware services. However, they are not sufficient to provide the multi-sensory media service. It is necessary to develop the novel ontology model that includes not only user context knowledge but also multimedia content knowledge, service context knowledge and sensory device knowledge. Also, the ontology model has to allow for an expansion possibility and an inference possibility. In this paper, we proposed the new ontology model considering contents and devices for the service. 3. Multi-Sensory Media Service The multi-sensory media service has the overall architecture as shown in figure 1. As shown in figure 1, the service provides users with sensory effects generated by sensory devices and audio-visual contents. In order to provide the service, first, a fonnal communication protocol must be defined in order to communicate between the player and sensory devices. Second, the standardization of stimulus which makes people feel realistic must be defined. Based on the standardization, many sensory devices can stimulate people to feel realistic. Third, knowledge which represents infonnation about multimedia contents and sensory devices is necessary. According to the given multimedia contents infonnation, the player defines sensory effects synchronized with the contents and sensory devices provide people with the sensory effects. Although not representing in figure 1, the context ofa user or environment may be considered because of the nature ofthe service. The service is not a predefined simple service but a flexible complex service based on the infonnation of multimedia contents and sensory devices. Also the service consists of different combinations of sensory devices and provides various multimedia contents. Considering the characteristics of the service, the knowledge which has the following three features is necessary for the service. First, the knowledge has to be interpreted semantically for the semantic interoperability among different devices. Second, the knowledge has to be standardized for various devices to share and exchange the knowledge each other. Last, the knowledge has to be possible to infer new infonnation. Because the knowledge cannot prepare infonnation for all situations, the service must be possible to infer new infonnation based on the usable knowledge. :lick wiad.... i8ccae. Figure 2. Ontology-based MUlti-Sensory Media Service Therefore, in this paper, we propose an ontology-based semantic knowledge representation as a method of representing infonnation for the multi-sensory media service. Figure 2 shows the ontology-based service architecture. In figure 2, the Ontology means the semantic representation about sensory effects and multimedia contents. The Device Ontology means the semantic representation about sensory devices installed in the service environment. The Inference Engine means the mechanism to infer necessary infonnation from given infonnation. ISBN Feb , 20091CACT 2009

3 In summary, an ontology is suitable for representing knowledge necessary for the multi-sensory media service because ofthe following advantages: - A formal and explicit knowledge representation - A machine-readable semantic knowledge representation - Knowledge sharing between heterogeneous systems - Easy knowledge expansion - Knowledge reuse and logical inference based on a semantic structure Figure 3 and figure 4 show the physical experiment environment for the multi-sensory media service. To stimulate user's five senses, we constitute various devices such as a display for video, a 5.1 channel speaker for sound, a light for brightness, a heater for temperature, a fan for wind and a perfume spray for smell. Practically, we constructed the system which satisfies user's senses except for taste. and their relationships. Also we added concepts for the sensory effects to the Ontology. haselcmcnt Macc hastirne Figure 5. Scene Class I CIIIS Property The Scene Class represents the description of multimedia contents. The information for the multi-sensory media service is organized by the unit of a scene. The Scene Class has relations with different classes such as, and as shown in figure 5. In otherwords, The Scene Class includes the information of place, time and elements predefined by a content producer. c:::> Clus ---+ Property.. Subclus.,... Figure 3. Physical Environment for the Multi-Sensory Media Service Figure 4. Physical Environment for the Multi-Sensory Media Service 4. Ontology The Ontology represents the information of sensory effects synchronized with multimedia contents. Appendixes show the whole structure of the Ontology. Before representing the sensory effects, the content information must be defined ahead. A scene means the minimum unit of multimedia contents like movies. A producer puts content into a movie by the scene. Each scene has basically several components such as place, time, actor, object, event, and so on. Therefore, in the Ontology, we defined the concept ofa scene and then described many concepts included in scenes The Class represents the place information about each scene ofmultimedia contents. In this case, the role ofthe Class is to define influences on Heating and Cooling effects of each scene. Although the Class meant location information in many different ontologies, we designed five subclasses ofthe Class according to climate as shown in figure 6. Therefore, we can define influences on the Heating and Cooling effects according to the climatic characteristics of places. The Class also represents the time information about each scene ofmultimedia contents. The time information also has influence on the sensory effect. The Class has the Season Class and the Day Class as subclasses as shown in figure 7. The Day Class has the following individuals: dawn, morning, forenoon, noon, afternoon, evening and night. The Season Class has the following individuals: spring, summer, fall and winter. I::? CT:::? 0 Cv Figure 7. Class Clus Subclus The Class represents the subject of sensory effects, for example, Sun, Wind, Water, Fog, and so on. In otherwords, we define sensoryeffects for representing element information ISBN Feb ,2009 ICACT 2009

4 of each scene predefined by a content producer. Figure 8 shows the structure ofthe Class. 5. Device Ontology Figure 8. Class -+ Property Subdus The Class has the effect information related to each element by the has relation and direction information by the hasdirection relation. In figure 8, the dotted classes represent the classes that are not fully considered in this paper. The Class represents the sensory effects to stimulate people. The information of the Class is inferred by the information of a scene. Figure 9 shows the structure of the Class. In figure 9, the dotted classes represent the classes that are not fully considered in this paper. We can define each effect according to each element. Each effect has attributes defined by information belonging to each scene. The Class represents the attributes what effects can have. We defined the following attributes: BreadthHorizon, BreadthVertical, Color, Frequency, RotateHorizon, RotateVertical, and Temperature. The BreadthHorizon means that an effect spread horizontally. The BreadthVertical means that an effect spread vertically. The Color defines a color of an effect. The Frequency defines the number ofoccurrences ofa repeating effect per unit time. The RotateHorizon means that an effect rotates horizontally. The RotateVertical means that an effect rotates vertically. The Temperature means that an effect changes circumference temperature. " ) e ':-mm :.. baadinldion---l cdj:::::: D1RECno LIGHT \ 1 NTESSITI ealj.: CtmA:> Figure 9. Class <::> c. 0 Value -+ Property... SubclaM Also, the Class can have direction and intensity information. The direction of an effect is inferred from the direction of an element defined by a content producer. The intensity of an effect is defined as the relative value inferred from information belonging to each scene. The Device Ontology represents the information of sensory devices generating sensory effects in the multi-sensory media service. Appendixes show the whole structure of the Device Ontology. We defined the Device Ontology in order to describe each sensory device. In other words, the Device Ontology defines characteristics of sensory devices installed in the service environment. The Device Ontology shares the Class and the Class in the Ontology. ATTRIBL'TE hudircdion...i 1 DIRECTION hasefrect Figure 10. Device Class o claas Value -+ Property Figure 10 shows the structure of the Device Class. The Device Class has the information of installed position in the service environment by the hasdirection relation. Also, the Device Class has the information of possible generating sensory effects and effect attributes. 6. Ontology Development and Evaluation The Web Ontology Language (OWL) is a language for defining and instantiating Web ontologies. In order to expand the limited expressiveness ofrdf Schema, a more expressive OWL has been defined by the World Wide Web Consortium (W3C) [8]. So far, OWL is the most widely used and outstanding ontology language in terms ofexpressiveness and reasoning capabilities. That's why we selected OWL among many ontology languages. In order to build OWL-based knowledge, we used Protege 3.4 [9] as an ontology editing tool. Protege is a free, open source ontology editor and knowledge-base framework. Protege supports GUI environments for building ontologies. As ontologies are widely used in a variety ofareas, ontology evaluation has become an important issue. The goal of an ontology evaluation is to reuse a useful ontology which defines correctly an interesting domain. To evaluate an ontology, the following criteria were identified: consistency, completeness and conciseness [10]. First, consistency means whether contradictory knowledge cannot be inferred from other axioms. Second, completeness means whether knowledge can represent explicitly a related domain. Last, conciseness means whether unnecessary redundancies or definitions can exist. In order to prove our ontology, first, we utilize Protege [9] with a consistency evaluation tool. Protege also supports the correctness ofsyntax and logic in building ontologies. ISBN Feb , 20091CACT 2009

5 Second, we show the result of various scenarios. Scenarios consist of different scenes of multimedia contents. We explain the inferred each effect information from each scene. The experiment scenarios are as follows. The following figures are screenshots of scenes. Each table shows the inferred information according to predefined information in each scene. Figure 11. Dessert Scene Table 1. Semantic Representation of the Dessert Scene Dry Noon Sun Heating Temperature Light Color Figure 12. Evening Glow Scene Table 2. Semantic Representation of the Evening Glow Scene Temperate Evening Sun Light Color BreadthHorizon Tornado Fan RotateHorizon RotateVertical As shown in above tables, we can confirm that proper sensory effects are inferred by the proposed ontology model. Our ontology model can fully represent multimedia contents and sensory devices for the multi-sensory media service. Also, ifnew contents and devices are added, the ontology model can be easily expanded by adding just related information without its structural change because of the model design consideration. Therefore, a content provider can determine the sensory effects synchronized with contents by defining simple scene information in order to provide the multi-sensory media service. 7. Conclusion We discussed a framework for the Multi-Sensory Media Service. And then, we introduced our ontology model for the service. The proposed ontology model consists of Ontology and Device Ontology. The Ontology represents semantic information about multimedia contents and sensory effects synchronized with the contents. The Device Ontology represents semantic information about sensory devices generating sensory effects. With our ontology model, the noble service can make users feel realistic and entertain users. With the proposed ontology model, we infers sensory effect information from the given multimedia contents information. In order to evaluate the proposed ontology model, we demonstrated various scenarios. Each scenario is a scene of a multimedia content such as a movie. After we defined the semantic information about the scenes, we confirmed inferred results. With scenario-based experiments, we proved the effectiveness of the proposed ontology model. Therefore, by describing the simple scene information, a content provider can define the sensory effects for the multi-sensory media service. 8. Future Work Figure 13. Snowfield Scene Table 3. Semantic Representation Frigid Morning Sun Temperate Afternoon of the Snowfield Scene Cooling Temperature Light BreadthHorizon BreadthVertical Color As a future work, we are planning to consider the context of a user and environment. The satisfaction of sensory effects may be under the influence of user preference and physical information of service environment, for example, the size of space, the type of space, and so on. Considering these factors, a personalized multi-sensory media service will be possible. Also, we will research about inferring alternative sensory devices for generating similar sensory effects when matched sensory devices do not exist. Figure 14. Tornado Scene ISBN Feb , 20091CACT 2009

6 ApPENDIXES REFERENCES [1] C. Timmerer. A Summary of the Representation of Sensory s(rose) [2] M. Weiser, "The Computer for the 21st Century," Scientific American 265, Nr. 3, S ,1991 [3] H. Chen, T. Finin, and A. Joshi. An ontology for context-aware pervasive computing environments. Knowl. Eng. Rev., vol. 18: ,2003. [4] X. Wang, D. Zhang, T. Gu, and H. Pungo Ontology based context modeling and reasoning using OWL. Pervasive Computing and Communications Workshops, Proceedings of the Second IEEE Annual Conference on: 18-22, [5] H. Chen, F. Perich, T. Finin, A. Joshi. SOUPA: standard ontology for ubiquitous and pervasive applications. Mobile and Ubiquitous Systems: Networking and Services, MOBIQUITOUS The First Annual International Conference on: ,2004. [6] E. Christopoulou and A. Kameas. GAS Ontology: An ontology for collaboration among ubiquitous computing devices. International Journal ofhuman-computer Studies, vol. 62: ,2005. [7] D. Heckmann. Ubiquitous User Modeling. PhD Thesis [8] D. L. McGuinness, F. Harmelen, "OWL Web Ontology Language Overview," W3C Recommendation, [9] Protege, [10]A. Gomez-Perez, "A Framework to Verify Knowledge Sharing Technology," Expert Systems with Application. Vol. 11, N. 4. pp , The Ontology The Device Ontology ISBN Feb ,2009 ICACT 2009