An Output Schema for Multimedia Data in Multimedia Database Systems

An Output Schema for Multimedia Data in Multimedia Database Systems Thomas Heimrich Technical University of Ilmenau, Databases and Information Systems, D-98684 Ilmenau thomas.heimrich@tu-ilmenau.de Abstract. Multimedia data differ significantly from alphanumeric data. The semantic of multimedia data depends from the data presentation. Up to now it is only possible to model structure and behaviour of multimedia data in a schema of a multimedia database. This paper proposes a new concept, called output schema. The output schema can be integrated into the multimedia database schema. An output schema is a description of the multimedia data output. It can be reused easily and adapted. The multimedia database can use the information from the output schema to optimize the data output and the data storage. Keywords: multimedia databases, modelling data output 1. Introduction Multimedia database management systems are mostly not completely new developed systems. Usually already existing object-relational or object-oriented database systems are used to store multimedia data. A multimedia layer is built on its top. A multimedia database must support different multimedia types (e.g. image, video, audio, fulltext) as basis data types. The database must offer also efficient storage structures and indices for these media types. Some extensions of commercial databases (e.g. DB2, ORACLE) offer these capabilities. It is possible to build complex multimedia documents from these basis media types. The complex multimedia types are defined in the structure and behaviour schema of a multimedia database. The structure schema consists of different structure types. A subtype-relation is defined on these structure types. With that a IS-A-Hierarchy can be build by means of inheritance. The possibilities of the inheritance concept are very useful for modelling complex multimedia data. The behaviour schema defines a set of methods. These are assigned to structure types. Usually it is desirable for behaviour inheritance to process a IS-A-Hierarchy of types. In this case it is necessary to allow that only methods in subtypes may be specialized, that is overloading is only allowed with compatible signatures (contravariance). The data output is very important for the semantic of the multimedia data. The semantic of an audio for example gets lost if its presentation speed is too fast. Up to

now it is only possible to model structure and behaviour of complex multimedia documents in a type system. We introduce the new concept of an output schema. It is supposed to model relationships during the data output. It is a description of the data presentation. Instances of this description are specific presentations. Section 2 gives a survey over some related work. Section 3 introduces the output schema and its requirements. Section 4 gives a notion about how the output schema may be implemented. Section 5 summarizes and concludes this paper. 2. Related work The need for modelling multimedia data representation is known since computer can handle multimedia data [5]. One of the most popular presentation languages is SMIL [8]. The multimedia data used by a SMIL presentation are stored as files in a filesystem. The filename is used to refer to multimedia data in a SMIL-Script. A SMIL presentation is inflexible. It is for example not possible to use the same SMIL presentation for picture Image_A and later for picture Image_B. The reuse of the structure defined by the SMIL-Script is not possible. If data are changed there is no way to announce these changes automatically to the SMIL-Script. Thus the SMIL- Script can not adapt its definitions according to these changes. The changes do not take into account what is defined by the SMIL-Script. An algebra for creating and querying multimedia presentations is described in [1]. The multimedia presentation algebra (MPA) contains generalisations of select, project and join operations in the relational algebra. This algebra operates on trees whose branches reflect different possible presentations of a presentation description. Creating a presentation means to create an instance of a presentation description. How to describe a presentation is not part of that paper. Candan et. all [6] have developed a view management for multimedia databases. They introduced virtual objects with associated spatial and temporal presentation constraints. Materializing a dynamic multimedia view corresponds to assembling and delivering an interactive multimedia presentation according to the visualization specification. The paper describes mainly the personalisation of multimedia views. Boll et. all [3, 4] have developed the ZyX data model for multimedia documents. It is a presentation-neutral description of a multimedia presentation. A presentation in the ZyX model is also a tree of specific presentation nodes. This model was implemented with a object-relational database. Classes were developed to model the presentation nodes. An disadvantage of this model is the small support for temporal relationships between multimedia data. Only the parallel and sequential presentation of media objects is supported. If an object model is used then it is impossible for the database to optimize the data output. Up to now a presentation description is not part of the multimedia database schema. Thus the database has no knowledge about the desired data output streams and the output conditions. It can not optimize the multimedia data output nor ensure the consistency between the presentation description an the stored multimedia data.

3. Output schema Section 3.1 defines some requirements which an output schema should fulfill. Then a formal definition for an output schema is proposed. Next the inheritance for output types is introduced. 3.1. Requirements on an output schema The output of multimedia data is more complex than the output of alphanumeric data. There are synchronisation relationships that have to be taken into account. During the output a certain quality of the data has to be guaranteed. Otherwise multimedia data might be semantically falsified. Up to now multimedia database systems were often used for the storage of multimedia data independent of each other. The data themselves are stored but only very few relationships between these data. It is assumed that the user builds the relations between the multimedia data through a database query. A query could be: Search Video1 and Audio1 and output them parallel. In [7] a formal query language is proposed for that. Constrained queries can be built. In these queries it is possible to specify temporal and spatial relationships (e.g. start(video2) end(video1) + 0.01) for the data presentation. The original query will be enlarged with a conjunction of these constraints. A corresponding enlargement of SQL seems to be easy to realize. But the following aspects are problematic: The user has to describe a complex output in the query. The specification of the output can only be used once. If the same output is desired again, the whole specification has to be built again. The user has to determine the temporal/spatial semantic of the output. There are no synchronisation relationships stored in the database. Thus the semantic of the multimedia data may be lost. We believe, that the presentation descriptions must be part of the multimedia database schema. The reasons for that are: Multimedia documents consist of multimedia data and complex spatial/temporal relationships between them. Without storing a presentation description it is not feasible to store and restore multimedia documents. The presentation of multimedia data is very important for their semantic. The resolution of images for example has high influence on their semantics. It has to be specified for the data output. Otherwise it may occur that the user sees the shown image with a too low resolution. Thus he can not see all the information of the image and he gets semantically incomplete or wrong data. The database needs the information about the desired presentation of the data in order to optimise the data output and the used data organisation. For example a specific video shall be presented parallel to a specific audio. Now it is important that the database chooses a storage organisation that allows the parallel output of the video and audio.

It is necessary to model the data presentation in a multimedia database. The model must have the following criteria (q.v. [3]): Reusability. Reusability of presentation description should be supported along two dimensions. Firstly reusability means that a presentation description can consist of other presentation descriptions. Secondly it means that from a presentation description many presentations can be built. A presentation description is a template for presentations. Adaptation. Extensions to and chances of presentation descriptions should be easy to do. Users should be able to adapt existing presentations according to their needs. Presentation-neutral Representation. The presentation description must be independent of the format used by the presentation. Therefore a multimedia database can support different presentation formats. The presentation-neutral description has to be converted into a presentation-specific format which is used for the playout of the multimedia data. It is desirable that this conversion is lossless. 3.2. Formal definition of the output schema Now the concept of the output schema is introduced. That is how model the data output of multimedia data. Output types are defined. They can be used as a reusable, adaptable and presentation-neutral representation for the data output. The output type is created by the user once and thereafter managed by the database. When querying the database, a particular output type can be used. It is not necessary to describe a complex output in the query. The output type is reusable and thus can be used for the optimization of the data output. A formal definition of a output type system (OT) is given as follows: (i) video, audio, image, fulltext OT (ii) If ot1, ot2 { OT { image, fulltext}}, TC={before, meets, overlaps, during, starts with, finishes with, equals} and tc TC then ot1 tc ot2 OT (iii) If ot1,ot2 {OT - audio}, SC={left, right, over, under} and sc SC then ot1 sc ot2 OT (iv) If ot1,ot2 OT, then ot1 substitute ot2 OT (v) if ON i are different names for output types and oti OT, 1 i n, then [ ON1 : ot1,..., ONn : otn] OT (tuple of output types) (vi) UDOT OT, UDOT is a set of names for user defined output types. The basis output types are defined in (i). These are the basis multimedia types which should be supported by a multimedia database. That is why these multimedia types should have a default presentation. Temporal output constraints are defined in (ii). The Allen-Relations [2] are used. It is also possible to use other concepts concerning temporal relations, for example difference constraints The important aspect here is that by means of combination any complex output type can be created. Temporal relationships can be defined only

on multimedia types which have an temporal dimension. Image and fulltext do not have this dimension. Thus it is not possible to define an output type with temporal relationships for these multimedia types. Spatial constraints are defined in (iii). Thus it is possible to specify also spatial relationships between multimedia data (video, image, fulltext) in an output type system. It is not possible to define spatial relationships for the multimedia type audio. This type has no spatial dimension. With (ii) and (iii) the synchronisation relationship between multimedia objects is modelled. Synchronisation is a special relationship that exist only between multimedia objects. In many cases the output of media objects should take place in a specific temporal and/or spatial order. Synchronisation relationships define the temporal/spatial data flow of the output. Another relationship between multimedia data is the substitution relationship. The same information can be represented by different media (e.g. speech or text). The designer of a media object decides which kind of representation he prefers (for example speech). He can allow another form of presentation as an alternative (for example text). The substitution relationship is defined in (iv). Point (v) defines a tuple type for output types. Thus it is possible to define smaller parts of a complex output type and give them a name. Through these names these parts can be used within the complex output type definition. An example for a tuple type is: [c1: video1 before video2, c2: video3 after video4]. The tuple type defines an array (tuple) of constraints. Every constraint has a name. It is assumed that a presentation build from a tuple output type has to satisfy all constraints which are defined in the tuple type. User defined output types (UDOTs) are defined in (vi). A user defined output type can be called from other output types through its name. We can use these types for building complex output types. An output type defines a template for a set of specific presentations. Assume the definition of the following UDOT: myudot: video equals audio. The output type has the name myudot. Through this name the output type can be used e.g. in queries. Here the output type defines the temporal ordering of multimedia data, that is a video and an audio have to be presented equally. Every presentation that presents a video and an audio equally is an instance of this output type. The output type can be reused with different specific multimedia objects. We used a presentation-neutral representation for output types. This representation can be transformed into different presentation languages. Through that the multimedia database can offer format independent presentations. 3.3. Inheritance for output types It should be possible to order output types in a output-type-hierarchy (IS-Arelationship). By that output types can be specialized. Existing output types can be easily re-used and adopted. An output type that outputs ot1 before ot2 can be specialized to ot1 two seconds before ot2. By means of derivation also substitutions between multimedia data can be supported. The output type video equals audio could be specialized to video equals fulltext. This output subtype

allows the parallel output of video and audio and as an alternative the parallel output of video and fulltext. It is possible to build output subtypes from all output types by means of derivation. A output type ot is output subtype of output type ot if every instance of ot is also an instance of ot. An instance of an output type is a specific presentation. For example an output type is specified as video equals audio. Every presentation in which a specific video is played out equal to a specific audio is an instance of that output type. We introduced a relation for output subtypes. For example the output type video two seconds before audio is an output subtype from the output type video before audio. All instances from the output subtype are also instances from its parent type. An output subtype has stronger output conditions than its parent type. The number of instances of an output subtype is smaller than those of its parent output type. The subtype-relation, (i) ot ot for every ot OT OT OT is defined as follows: (ii) [ 1': ot1',..., OTm': otm' ] [ OT1: ot1,..., OTn : otn] OT if (a) m n and (b) ( OT i,1 i n)( OTj',1 j m) OTi = OTj' ( otj' oti ot i substitute ot j ) The subtype relation for tuple output types is defined in (ii). It is assumed that a presentation to a tuple output type must hold all defined constraints Thus point (ii)(a) defines that an output subtype can have more output constraints than its parent output type. For example the output type [c1: video1 before video2, c2: video3 after video4, c3: video1 equal audio1] can be a subtype of [c1: video1 before video2, c2: video3 after video4]. The output types defined in the output subtype have to be more specific than the output types defined in the parent output type. This is defined with (ii)(b). The output type [c1: video1 3 seconds before video2, c2: video3 after video4] can be a subtype of [c1: video1 before video2, c2: video3 after video4]. This example also shows why the conditions in a tuple output type must have names. Only through these names it is possible to determine whether a constraint is more specific than its definition in the parent output type. Through the inheritance of output types it is easy to adapt existing output types. Existing output types can be easily reused. 4. Implementation 4.1. Definition of output constraints In order to model multimedia data it is necessary to model their structure, behaviour and presentation. A concrete class definition must consider all these parts. An example for a class definition is:

Class example{ // structure type NameOfVideo video; NameOfAudio audio; // behaviour type setnameofaudio(a audio); // output type NameOfVideo equal NameOfAudio } To keep it simple we used pseudo code for this example. To create a specific database schema a data manipulation language has to be used. The novelty here is, that a class describes its own presentation and that the output types of classes can build a IS-A-Hierarchy (section 3.3). Output schema and structure schema are very similar. The structure schema in a object-relational databases defines the schema of typed tables. The output schema defines the presentation of all entities in these typed tables. If an user asks for entities from that table the database knows how to present the result. The database designer can determine how multimedia data should be presented. The output schema is defined in a very abstract way. In a concrete implementation a component in the database is needed which converts these presentation-neutral representations into a specific presentation language (e.g. SMIL). The user sends a query to the multimedia database and gets e.g. a SMILscript and the required data as a result. A presentation client (e.g. RealPlayer) is used to show the multimedia presentation what represents the final result of the database query. 4.2. Checking the output constrains The shown way to define output constraints is very simple for the database designer. It is very hard to check these constraints in that form. From that we transform the Allen-Relations into different constraints. As an example the constraint video equal audio can be written with different constraints as follows: start ( audio) start( video) 0, start ( video) start( audio) 0 end ( audio) end( video) 0, end ( video) end( audio) 0 It is easy to see that a large set of different constraints can caused by some simple Allen-Relation. A constraints graph can be build from different constraints. With a constraint graph the set of different constraints can be resolved in polynomial time. There is no conflict in the defined set of different constraints if the constraint set is solvable. Furthermore the result for the constraints set can be seen as a schedule for the data output. Different constraints can easy be checked during the data output. The concrete time points for the start and the end of media objects can be used in the different constraints.

4.3. Using an output schema Often users do not ask for complete objects from a specific class. In ad-hoc queries users ask for attributes from different tables. A presentation description for that kind of query is necessary. It is possible to define output types as user defined output types (UDOT). Those are independent from the structure and behaviour schema. The user can define a output schema in the way it is shown in section 3. A user can build arbitrary output types in the database. In a query these output types can be used instead of describing the complete presentation in the query. It is important that the attributes which should be presented must match the parameters of the used output type. An example is: SELECT video1, audio1 FROM tablename WITH OUTPUT TYPE myudot(video1,audio1) Thus it is possible that a user defines its own output types. He has not to use the output types which are defined by the database designer. Still a multimedia database must also have the capability to describe the complete presentation in the query. Nobody is able to predict every possible kind of the presentation at modelling time. 5. Conclusion When looking at multimedia data it is not automatically known how to present these data correctly. Thus a model for the structure, the behaviour and the output is needed. This paper introduces the concept of the output schema. Thus it is possible describe the presentation of multimedia data. A subtype relationship for output types was also defined.. So a IS-A-Hierarchy for output types can be build. The output types are a reusable, adaptable and presentation-neutral representation for the data output. A output type can be part of a class definition. Thus the developer of the class can define the presentation of the multimedia data. A output type can also be independent from structure and behaviour schema. A user can define its own output types. This kind of output types can be used within a database query. Thus a definition of complex presentation constrains in a query is not necessary anymore. A big advantage of the proposed output schema is its integration in the database schema. Through that a multimedia database can optimize the data output and the used data structures. Furthermore it is easy to see that presentation constraints can be checked by the database. If an attribute from type video e.g. changes its length then it is possible to check all output types for that attribute. The database can determine the invalid output types. The concept of the output schema is very similar to structure and behaviour schema. Thus it is easy to integrate the output schema into existing object-relational or object-oriented databases.

References [1] Adali S., M.L. Sapino and V.S. Subrahmanian: An algebra for creating and querying multimedia presentations. Multimedia Systems, 8:212-230, 2000 [2] Allen J.F.: Maintaining Knowledge about Temporal Intervals. Communications of the ACM, 26(11):832-843, 1983 [3] Boll S. and W. Klas: ZyX - A Semantic Model for Multimedia Documents and Presentations. In: Semantc Issues in Multimedia Systems, pages 189 209, 1999. [4] Boll S. and W. Klas: ZyX A Multimedia Document Model for Reuse and Adaption of Multimedia Content. TKDE 13(3): 361-382, 2001 [5] Christodoulakis S. and Analyti A.: Guest Editorial Special Issue on Multimedia Information Systems. IS 20(6): 443-444 (1995) [6]Kasim S. C., E. Lemar and V.S. Subrahmanian: View Management in multimedia databases. The VLDB Journal, 9(2): 131-153, 2000 [7] Marcus S. and V.S. Subrahmanian: Foundations of Multimedia Database Systems. Journal of the ACM, 43(3): 474-523, 1996 [8]W3C: Synchronized Multimedia Integration Language (SMIL 2.0). http://www.w3.org/tr/smil20, 2001