Abstract. Proceedings of the IEEE Sixth International Symposium on Multimedia Software Engineering (ISMSE 04) /04 $20.

Transcription

1 Spatial and Temporal Reasoning in Multimedia Information Retrieval and Composition with XDD Napat Sukthong CS&MIS Program, Faculty of Informatics, Mahararakham University, Mahasarakham THAILAND Abstract This research first proposes a general framework for retrieval and composition of spatial and temporal multimedia. It is based on an Equivalent Transformation theory and an XDD theory. It employs the MPEG-7 and SMIL standards which provide the tools to annotate the spatial and temporal relations inside multimedia and the tools to compose the multimedia, respectively. SMIL is selected to be one of the representation languages for the multimedia documents and presentations. Yet this has no enough efficiency to handle higher-level spatial and temporal relations, whereas the MPEG-7 provides tools to annotate those kinds of relations. These tools are used to decompose multimedia contents into segments and synchronize them together. Nevertheless, they should have more expressiveness power for retrieval and composition. As a result, the XDD theory is employed to enhance the language in terms of expressibility and computability. Moreover, by making use of XDD, constraints and axiom specifications are easily defined. This paper also defines a modeling language called SMILD that be able to represent spatial and temporal multimedia. 1. Introduction At the present, the amount of multimedia information is increasing dramatically and the interest in it is pervasive. Entertaining and educational groups are exploring new multiple applications while, the computer, telecommunication and its electronic-consumers, industries are actively developing underlying technologies. Multimedia Document (MD) refers to the integration of several media (such as text, image, audio, video and animation) into a rich complex structured document, between the components of which exist temporal, spatial, structural and semantic relationships. Hence, an MD model integrates media description models with spatial and temporal models. As a result, modeling MDs differs from modeling conventional data types (e.g., data in record form) due to their special features. In consequence, Multimedia Information Retrieval (MIR) also differs from that of textual information. Furthermore, MIR involves not only the syntax and the rich semantics of MDs, but also their spatial and temporal relations. Consequently, many developments of appropriate and efficient generic multimedia information retrieval systems have been proposed, including Context-based retrieval, Content-based retrieval and Semantic-based retrieval. Automatic multimedia retrieval tools provide efficient means for navigation. Even though traditional methods allow users to raise queries and obtain the corresponding results, retrieval accuracy is severely limited. This is because of the limited ability of users in describing exactly the inherent complexity of multimedia. Presently, there are rarely systematic means available by way of systems which manage spatial and temporal multimedia retrieval. This research employs these relations as constraints of the query. At the present time, little research has dealt with the capability of spatial and temporal multimedia retrieval. In what follows, the rule-based programming language, XET [3] is employed while dealing with spatial and temporal relations. Synchronized Multimedia Integration Language (SMIL) [4] is a W3C s recommendation, which has recently emerged as a standard for encoding multimedia presentations for delivery over the Web. It is a collection of Extensible Markup Language (XML) elements and attributes, which can be used to describe the spatial and temporal coordination of one or more media objects. Moreover, SMIL (as a W3C recommendation and not a proprietary technology) does not tie the implementation of a synchronized presentation system to particular platforms or programming languages. In SMIL, it can specify subparts of media in terms of a time instant from the beginning of the media. But it is a rather low level and limited mode of specification of media subparts. This leads to a lack of support for search facilities via search term entry. In order to utilize structured media whose information content is described at a higher level and make it available during the composition process, MPEG-7 [9] is selected to be a standard format for content information description. Similar to those in MIR, there exist many attempts at seeking some appropriate approaches to integrate multimedia objects. This task presents several technical challenges, which, in particular, are related to taking spatial and temporal relation constraints into account such as the development of standard means for specifying and enforcing spatial and temporal relations between various media objects within multimedia documents.

2 The three promising areas of research, mentioned above, indicate the need for some standardized tools for the description of multimedia content in the multimedia environment. Furthermore, there is a lack of ability to query the multimedia documents by efficient specification of spatial and temporal relations within multimedia documents. Besides, there is a lack of the ability of on-demand integration of retrieved multimedia objects (with respect to spatial and temporal constraints) in order to create a new multimedia presentation. As a result, this research introduces a framework that enables E-learning by utilizes the metadata and the underlying theories e.g. XDD, ET and Spatio-Temporal relationships. In the proposed framework, XDD serves the three important roles: (i) multimedia-content represent language, (ii) application-rule language, and (iii) query or service-request language. The remainder of the paper is as follows. In Section 2 describes the underlying concepts, theories and tools. Section 3 describes the spatial and temporal representation. Section 4 describes the retrieval and spatial and temporal multimedia. Section 5 describes the composition of spatial and temporal multimedia. The paper concludes in Section 7 with the addition evaluations and anticipated the future work direction. 2. Background 2.1 Temporal Relations Temporal relations describe the relationship between two multimedia objects with respect to time. The topic of relations between temporal intervals has been addressed in [2], which contains a definition of a complete set of temporal relations between two actions. These relations are: before, during, overlaps, starts, ends, equal and their inverses (does not apply to equal). In analogy to the instant-based situation, there exist 2 13 = 8192 indefinite interval relations. However, knowledges about how to determine temporal relation in an implicit manner must be considered. Suppose that there are three or more multimedia objects, how is one to represent all possible temporal relations among all of these multimedia objects? For example, A before B, and B equal C, which could determine that A before C (according to the transitivity property). 2.2 Spatial Relations Spatial relations describe the relationship of two multimedia objects with respect to space at certain points in time. Spatial relation can be grouped into: a directional relation (south, north, west, or east), a mixed directional relation (northwest, northeast, southwest, or southeast), or a positional relation: left, right, above, below, ontopof, or under. Similar to those temporal relations, knowledges about how to determine the spatial relation in an implicit manner must be considered. Compared to temporal relation, when the query is input, it usually retrieves the desired multimedia data by using spatial relationship. For example, one might want to retrieve the scene of boxes containing a ball, and a doll from a multimedia database. Those spatial relationships are binary relationships, such as A contains B, where the operands A and B are source and target objects, respectively. 2.3 Equivalent Transformation Theory Equivalent Transformation (ET) [1] is a new computational paradigm based on semantic preserving transformations. A given declarative description P is transformed successively and equivalently into P 1, P 2 until a desirable declarative description P n is obtained. Computation under the ET paradigm is possible through Equivalent Transformation Compiler (ETC) and Equivalent Transformation Interpreter (ETI). 2.4 ETI: Equivalent Transformation Interpreter. ETI is interpreter system (virtual machine) that executes programs written using ET rules. ETI has been designed to be powerful, reliable and extensible. Its features include automatic memory management, dynamic addition/deletion of ET rules, extensible built-in predicates, multi-head ET rules, Internet access and more. 2.5 ETC: Equivalent Transformation Compiler. ETC is a highly efficient implementation of ET programming language. It compiles ET programs into C codes, which is then compiled to native machine executable, resulting in a fast program execution. 2.6 XDD Theory XDD [11] is a language which enables representation of the semantics of a Web resource. It is founded on a theoretical basis upon which representation and computation of as well as reasoning with XML data can be carried out in a uniform and succinct manner. It employs XML syntax as its underlying data structure, and enhances XML s expressive power by employment of a Declarative Description (DD) theory. Since Resource Description Framework (RDF) s exchange-format is based on the XML syntaxes, XDD can immediately process RDF data. Computation with XDD is carried out by means of ET. A given XDD P is transformed successively and equivalently into P 1, P 2 until a desirable XDD P n is obtained. Computation under the ET paradigm is possible through ETC, ETI and XML-Equivalent Transformation Compiler (XET). 2.7 XET XET [3] is a new programming language designed to work on a XML processing domain by employing XML syntax and ET computational paradigm. Hence, it processes the following properties: users can straightforwardly create and process XML documents, errors occurred during XML document generation can be substantially reduced, errors as well as costs of learning various Web-development tools and languages can be reduced, and XET programs can be easily integrated into the WWW.

3 Furthermore, XET incorporates a fragment of XML documents into its data types. As a result, users have full capability to manage any part of XML documents and can straightforwardly create and process their documents, it is based on rules and rule priority. It provides query facilities and supports also document processing. Moreover, XET provides a validation function which can be used to validate function which can be used to validate XML documents at run-time. Therefore, it can validate not only input documents, but also output and intermediate documents. 3. Spatial and temporal multimedia representation A multimedia document is represented under the MPEG-7 standard [9] (a standard for describing multimedia content). It provides the Multimedia Description Scheme (MDS), which comprises many tools for description of spatial and temporal relations. 3.1 Requirements of a modeling language to represent spatial and temporal multimedia There are numerous requirements for a language which facilitate syntactic means for establishing semantic descriptions of multimedia content. The language must be able to: support the definition of syntactic rules to express and combine description structures at various levels of details, state spatial, temporal and conceptual relationships between the components of a description and between descriptions, facilitate a diverse set of linking mechanisms between descriptions and the described data, which includes, in particular, means of segmentation for temporal media, and be platform and application independent and humancomprehensible and machine-readable. More crucially, in order to represent the spatial and temporal multimedia, a representation language must be able to: represent qualitative spatial and temporal relations (e.g. BEFORE, ABOVE ) inside multimedia, and represent quantitative spatial and temporal relations (e.g. TM00M05S10 ) inside multimedia. These two relations, as has been mentioned above, are called extensional data and relations data and relationships which are explicitly stored in multimedia. In contrast, spatial and temporal data are represented in an implicit manner inside multimedia. Thus, the language should be also able to: represent intensional data and relations which are those data and relations computed or deducted from extensional data and relations e.g., A before B, and B before C, this can be implied that A before C. 3.2 XDD: a modeling language to represent spatial and temporal multimedia Spatial and temporal information is represented in an implicit manner inside multimedia, whereas MPEG-7 cannot represent this kind of data and relations. Consequently, XDD [11] is selected to represent multimedia document, spatial and temporal relations, constraints and rules. In the XDD data model, extensional data and relations (static data and relations e.g., above, before etc) are represented by ground XML expression, while the intensional data and relations (data or relations deducible from existing data and relations e.g., A before B, and B before C, this can be implied that A before C, then A before C is called intensional relation) are represented by non-ground XML expression. Every component of an XDD expression can contain variables, i.e., tag name, attribute names and values, attribute-value pairs, sub expression and some partial structures. These expressions are represented by N-variables ($N), S-variables ($S), P-variables ($P), E- variables ($E), and I-variables ($I), respectively. For example, $E:expression and $S:value are E-variable and S-variable, respectively. XDD description is a set of XML clauses, each of them in the form: H <-- B 1, B 2 B n where n 0, H is an XML expression, B i is an XML expression or an XML constraint in which its order is immaterial. H is called head and B i body of an XML clause. If the body is empty, the clause is called an XML unit clause. This clause is denoted by H <- or simply H; it represents facts or XML elements. In contrast, if the body is not empty, the clause is called an XML non-unit clause. Moreover, an XML element or document can be mapped directly into a ground XML unit clause. In the proposed framework, XML unit clauses are the elements of an MPEG-7 multimedia document. Figure 1. A video used in every example in this paper For example, the following MPEG-7 multimedia document describes the information of each video segment in Figure 1: C G1 : C G2 : C G3 : <VideoSegment id= VS1 > <TemporalSegmentLocator id= TSL1 > <mediauri> <MediaTime id= MT1> <mediatimepoint>t00:00:00</mediatimepoint> <mediaduration>pt30s</mediaduration> </VideoSegment>. <VideoSegment id= VS2 > <TemporalSegmentLocator id= TSL2 > <mediauri> <MediaTime id= MT2 > <mediatimepoint>t00:00:31</mediatimepoint> <mediaduration>pt30s</mediaduration> </VideoSegment>. <VideoSegment id= VS3 > <TemporalSegmentLocator id= TSL3 > <mediauri>

4 <MediaTime id= MT3 > <mediatimepoint>t00:01:00</mediatimepoint> <mediaduration>pt30s</mediaduration> </VideoSegment>. C G4 : <VideoSegment id= VS4 > <TemporalSegmentLocator id= TSL4 > <mediauri> <MediaTime id= MT4 > <mediatimepoint>t00:01:30</mediatimepoint> <mediaduration>pt30s</mediaduration> </VideoSegment>. Furthermore, it can represent the relationship of each video segment. C G5 : <BinaryTemporalSegmentRelation Clause C G5, C G6 C G7 gives id= VS1 > information explicitly about <source idref= #VS1 /> the temporal relation. C G5 <target idref= #VS2 /> means scene Elephant </ BinaryTemporalSegmentRelation>. comes before scene Koala C G6 : C G7 : <BinaryTemporalSegmentRelation id= VS2 > <source idref= #VS2 /> <target idref= #VS3 /> </ BinaryTemporalSegmentRelation>. <BinaryTemporalSegmentRelation id= VS3 > <source idref= #VS3 /> <target idref= #VS4 /> </ BinaryTemporalSegmentRelation>. C G6 means scene Koala comes before scene Kangaroo. C G7 means scene Kangaroo comes before scene Monkey On the other hand, XML non-unit clauses are used to represent rules or knowledge. For example, the following example is the representation of the transitivity rule for before relation: C R1 : <BinaryTemporalSegmentRelation id=$s:segment_a> <source idref=$s:segment_a/> <target idref=$s:segment_b/> </BinaryTemporalSegmentRelation> <-- <VideoSegment id=$s:segment_a> <TemporalSegmentLocator id=$s:a_loc> <mediauri>$s:url1</mediauri> <MediaTime id=$s:a_mt> <mediatimepoint>$s:a_tp </mediatimepoint> <mediaduration>$s:a_md</mediaduration> </VideoSegment>, <VideoSegment id=$s:segment_b> <TemporalSegmentLocator id=$s:b_loc> <mediauri>$s:url2</mediauri> <MediaTime id=$s:b_mt> <mediatimepoint>$s:b_tp </mediatimepoint> <mediaduration>$s:b_md</mediaduration> </VideoSegment>, ADD($S:A_TP, $S:A_MD, $S:A_Result), LT($S:A_Result, $S:B_TP). The C R1 clause means that, if there is one video segment which comes before another one (overall duration of 1st segment less than the beginning time point of the 2nd segment), this relationship is called before. C R2 : <BinarySpatialSegmentRelation id=$s:id1> <name>over</name> <source idref= $S:var_A > <target idref= $S:var_C > </BinarySpatialSegmentRelation> <-- <BinarySpatialSegmentRelation id=$s:id1> <name>inside</name> <source idref= $S:var_A > <target idref= $S:var_B > <BinarySpatialSegmentRelation id=$s:id2> <name>over</name> <source idref= $S:var_B > <target idref= $S:var_C > </BinarySpatialSegmentRelation>. The C R2 clause means that, if there is an object (1st object) inside another object (2nd object), the latter object (2nd object) is over another one (3rd object); it can be described? By saying that, the 1st object is over the 3rd object. 4. Retrieval of spatial and temporal Multimedia Section 3 has discussed the specifications of a modeling language to represent spatial and temporal multimedia. This Section will address those of a query language and a computation model for retrieval of spatial and temporal multimedia. After that, the use of XDD as a computational model for retrieval of spatial and temporal multimedia is discussed A multimedia query language should provide predicates for expressing conditions on the attributes, the content, and the structure of multimedia objects. Structural and semantic predicates can refer to temporal or spatial relations of multimedia objects. Thus, a query language for retrieval of spatial and temporal multimedia must be able to: specify spatial and temporal relations for querying multimedia objects. In other words, the query language should be able to provide these kinds of query: temporal database query, spatial database query, and spatio-temporal database query, and provide spatial and temporal (computational) operators to retrieve multimedia objects. 4.1 Requirements of a query language for retrieval of spatial and temporal multimedia A multimedia query language should provide predicates for expressing conditions on the attributes, the content, and the structure of multimedia objects. In general, query predicates can be classified into three different groups of predicates [5]: attributes, structure, and semantic. Structural and semantic predicates can refer to temporal or spatial relations of multimedia objects. [8] addresses three crucial query specification issues in MPEG-7 document retrievals: The query language should be able to express extensional and intensional data and relationships in query language constructs. MPEG-7 documents often contain irregular document structures. In MPEG-7 XML documents, a Segment tag can also be inside another Segment tag. MPEG-7 content structures are based on their own datatypes and description schemes (DSs) rather than on XML element hierarchy. MPEG-7 XML documents normally are not data-centric documents which are collections of almost identical structures. As a result, a full document addressing query construct is needed to precisely specify the desired document locations in recursive or contextual XML structures for retrieving information.

5 Multimedia object descriptions have, by their nature, spatial and temporal synchronization constraints. The scheduled structures are inherently inside multimedia documents. Thus, a query language for retrieval of spatial and temporal multimedia must be able to: specify spatial and temporal relations for querying multimedia objects. In other words, the query language should be able to provide these kinds of query: temporal database query, spatial database query, and spatio-temporal database query, and provide spatial and temporal (Computational) operators to retrieve multimedia objects. 4.2 Requirements of a computation model for retrieval of spatial and temporal multimedia The previous section described the MPEG-7 description tools necessary to implement search functionality. This section describes the retrieval model for searching audio-visual material based on their associated MPEG-7 descriptions. Firstly, the requirements of methods for searching audio-visual data are examined in Section Requirements of a retrieval model for MPEG-7. MPEG-7 description schemes (DSs) define the schemes for representing structure, content and relationships of multimedia data; MPEG-7 DSs are specified as XML schemes. An MPEG- 7 description is an instance of a DS, so an MPEG-7 description is considered as an XML document. An XML document is a structured document, in the sense that the XML language is one way to capture the structure of a document. With this view in mind, the requirements for a model for structured document retrieval, and in particular, XML document retrieval, apply to MPEG-7 retrieval: Provide a relevance-based ranking function. Retrieve the most relevant document elements (an entire audio-visual sequence vs. a segment). Consider relationships (spatial, temporal, etc.) for representing (indexing) and retrieving MPEG-7 elements. Support query formulation with respect to content and attributes. Model content-bearing entities (segments) as well as semantic entities (persons, places, etc). Reflect the inheritance of attributes and elements. The requirement 1 holds for all Information Retrieval models, requirements 2 and 3 for structured document retrieval, requirement 4 is particular to XML retrieval, and requirements 5 and 6 are particular to MPEG-7. In this research, the requirements 1 and 2 are solved in the retrieval model, requirement 3 and 4 are dealt with XDD and lastly, requirement 5 and 6 have already been attended to under MPEG-7 specification. 4.3 XDD: the computational model for spatial and temporal multimedia retrieval A query about information in a multimedia document is formalized as an XDD description, comprising one or more XML clauses. For each query clause, its head describes the structure of the resulting XML elements. Its set of XML expressions in its body represents some particular XML documents or XML elements to be selected. The set of XML constraints describes selection condition. The set of setaggregations constructs sets or groups of related XML elements to be used for computation of summary information. This syntax intuitively separates a query clause into the three parts: Constructor described by the head, Pattern described by the set of XML expressions in the body, and Filter described by the set of XML constraints and set-aggregations in the body. Figure 2 shows the usage of XDD as a computational model in this framework Figure 2. Usage of XDD as a computational model for retrieval Furthermore, it provides spatial and temporal (computational) operators to retrieve multimedia objects. 4.4 Queries of spatial and temporal multimedia Temporal database queries Query1: Find any multimedia in which a scene of elephant comes before a scene of monkey. <BinaryTemporalSegmentRelation id=$s:id1> <source idref= $S:elephant_scene /> <target idref= $S:monkey_scene /> </BinaryTemporalSegmentRelation> Figure 3. Query in XDD for Temporal database queries Result1: (elephant) must appear before (monkey) but not necessarily immediately after (elephant) Spatial database queries Query2: Find any multimedia that has a koala above a tree

6 <BinarySpatialSegmentRelation id=$s:id1> <name>above</name> <source idref= $S:elephant_scene /> <target idref= $S:monkey_scene /> </BinarySpatialSegmentRelation> Figure 4. Query in XDD for Spatial database queries Result2: (koala) is above (tree). We know that this is the starting video clip of the query such that the rest of the multimedia presentation begins to display Spatio-temporal database queries Query3: Find any multimedia that has a scene of elephant comes before a scene of a koala above a tree <BinaryTemporalSegmentRelation id=$s:id1> <source idref= $S:elephant_scene /> <target idref= $S:monkey_scene /> </BinaryTemporalSegmentRelation> Figure 5. Query in XDD for Spatio-Temporal database queries 5. Composition of multimedia Section 4 has discussed the specifications of a query language to retrieve spatial and temporal multimedia. This section addresses those of a modeling language for composition of spatial and temporal multimedia. 5.1 SMIL SMIL provides the proposed framework able to integrate multimedia. It specifies how components relate temporally and spatially during a presentation. Furthermore, it is possible to specify sub-parts of media in terms of the instant from the beginning of the media. But it is a rather low level and limited way of specification of media sub-parts. This leads to a lack of support for dynamic multimedia composition. 5.2 XDD: a computational model for composition of spatial and temporal multimedia Nevertheless, the SMIL languages have no computational capability. Thus, one cannot pose any queries or compute spatial and temporal relations. Thus, XDD has been selected as the computational model for composition of spatial and temporal multimedia. It will map and retrieve results into SMIL format output by taking spatial and temporal relations into account. As described in Section 4, Clause C1 has the information about this multimedia, Clauses C2 to C5 represent notation of each multimedia object, Clause C6 to C8 show the knowledge about the relationships among multimedia objects. Then Rule R1 has been added. Rule R1 represents the knowledge that if A is before B and B is before C then A is before C. Then the query result from the above rule is:

7 Table 1. The core components of each system From the above result, it produces many related result. It is easy to add some constraints to filter the results, as well as the head part of XDD clause can be changed into SMIL format as follow. Query Q1 means select only scene in which object koala comes before other objects. The result can be seen as in Figure 6. It shows that the computation in this framework is more efficient and powerful. Furthermore, it provides semantic preservation. Rules and constraints are added to the framework easily and flexibly without affecting its other sub-systems. From the information addresses above, it found that the use of Dublin Core in DSTC-1 and DSTC-2 is too simplistic to infer interesting rich or interesting semantic relationships. It is not designed for describing multimedia resources. It is inadequate, in particular, for describing fine-grained details such as segmentation, formatting and low-level audiovisual feature metadata, which would be most useful for inferring interesting semantic relationships between multimedia objects. Furthermore, the usage of Cuyper as constraint handler for XML-based multimedia document is very difficult: When the user creates any constraints in ECLipSe or Logtalk; if there is any problem, it is not possible to efficiently revise constraints. The user has to rebuild all of them. In the Madeus system, Madeus language is used as a modeling language to represent multimedia. A problem arises when it transforms Madeus language into the input format-smil; the system cannot guarantee semantic preservation of the transformation. In summary, Table 2 concludes from the criteria stated: Table 2. Functionality comparison of systems Figure 6. SMILD Architecture 6. Evaluation In this section, the overall feature of the present systems; Madeus [7], Cuyper [10], DSTC-1[6], and DSTC-2[6] will be discussed. It can be classified into two headings: representation of spatial and temporal multimedia, and computation of spatial and temporal relations for multimedia retrieval and composition. (*) means with some limitations. 7. Conclusion The results of this research are the language that developed to represent spatial and temporal multimedia information, namely SMILD (SMIL + XDD), and the spatial and temporal multimedia retrieval and composition framework. H B 1,B 2, B n Constructor Pattern Filter XDD View MPEG-7 SMIL XML MM DB XDD Figure 7. SMILD Architecture Spatial Temporal Semantic SMIL has been selected to be one of the representation languages for multimedia documents and presentations, but it is lacking in an enough efficiency to handle higher-level spatial and temporal relations. Thus, MPEG-7 MDS has been selected to remove this weakness of SMIL. Hence, one obtains

8 a language, which is efficient enough to represent a multimedia document as well as multimedia presentation. Nevertheless, the two former languages have no computational capability. No queries can be posed or spatial and temporal relations computed. Thus, XDD [11] has been selected to be the computational model for the proposed framework and XET [1] to be the programming language for computing spatial and temporal relations. At this stage, there exists a new language with a high efficiency in the representation of spatial and temporal multimedia. The language has also an ability in spatial and temporal relation computations for multimedia retrieval and composition. This new language is called SMILD (SMIL+XDD). It is a language developed to represent spatial and temporal multimedia information. Figure 7 shows the architecture of SMILD. Figure 8 shows some contribution of the proposed framework: Representation of spatial and temporal multimedia Search and efficient retrieval of spatial and temporal multimedia by specification of temporal or spatial semantically relations inside multimedia as query constraints. Cooperation with spatial and temporal relations inside multimedia in representation and computation of new relations. This research is just the beginning of much possible research in this area. There are many areas in which temporal or spatial relations and multimedia information retrieval can be applied together. Meanwhile, the researcher plans to continue developing presentation-authoring system as well exploring issues such as indexing, distributed multimedia composition, and the researcher hope that his results encourage further research by other as well. 8. Acknowledgements This work was supported in part by Faculty of Informatics Research Fund, Mahasarakham University. 9. References [1] Akama, K., Shimitsu, T., and Miyamoto, E., Solving Problems by Equivalent Transformation of Declarative Programs, J. Japanese Society of Artificial Intelligence (JSAI) 13(6), 1998, pp [2] Allen, J. F., Maintaining Knowledge about Temporal Intervals, Communications of the ACM 26(11), November 1983, pp [3] Anutariya, C., Wuwongse, V. And Wattanapailin, V., An Equivalent-Transformation-Based XML Rule Language, Proc.Int l Workshop Rule Markup Languages for Business Rules in the Semantic Web, Sardinia, Italy, June [4] Ayers, J., Synchronized Multimedia Integration Language (SMIL) 2.0, World Wide Web Consortium Recommendation, [Jan. 2004] [5] Baeza-Yates, R., and Ribeiro-Neto, B., Modern Information Retrieval, ACM Press, Addison-Wesley, Harlow, [6] Hunter J., Little S., Building and Indexing a Distributed Multimedia Presentation Archive using SMIL", Proc. 5th European Conference on Research and Advanced Technology for Digital Libraries, Darmstadt, [7] Jourdan, M., Layaida, N., Roisin, C., Sabry-Ismail, L., and Tardif, L., Madeus, an Authoring Environment for Interactive Multimedia Documents, Proceeding of ACM Multimedia 98, Bristol, England, Sep 1998, pp [8] Liu, P., Hsu, L.H., Queries of Digital Content Descriptions in MPEG-7 and MPEG-21 XML documents, [May 2004] act/ html [9] Martinez, J., "Overview of the MPEG-7 Standard (version 9)", ISO/IEC JTC1/SC29/WG11 N5525, Pattaya, March [Jan 2004] 7/mpeg-7.htm [10] van Ossenbruggen J.R., Cornelissen F.J., Geurts, J.P.T.M., Rutledge, L.W., Hardman, H.L., Cuyper: a semi-automatic hypermedia generation system [April 2004] [11] Wuwongse, V., Anutariya, C., Akama, K. and Nantajeewarawat, E. XML Declarative Description (XDD): A language for the Semantic Web, IEEE Intelligent Systems 16(3), 2001, pp Figure 8. Multimedia retrieval and composition framework