Demonstration of an MPEG-7 Multimedia Data Cartridge

Size: px

Start display at page:

Download "Demonstration of an MPEG-7 Multimedia Data Cartridge"

Easter Thomas
5 years ago
Views:

1 Demonstration of an MPEG-7 Multimedia Data Cartridge Mario Döller, Harald Kosch, Bernhard Dörflinger, Alexander Bachlechner, Gisela Blaschke University Klagenfurt Institute of Information Technology 1. INTRODUCTION Multimedia Database Systems (MMDBMS) organize and store multimedia data for content retrieval. The relying multimedia data models, e.g., [9], represent abstractions of media objects for querying, indexing, and so on. However, most currently available implementations reval shortcomings. Either they are limited by one kind of multimedia data supported or by the capacity of their semantic modelling. Despite the upcoming MPEG-7 standard for representing low-level and high-level features of multimedia and respective annotation and use tools (see we are not aware of any MMDBMS product which integrates this standard for the purpose of a more meaningful indexing and querying. In this context, our paper presents a Multimedia Data Cartridge (MDC) for demonstration at ACM Multimedia 2002 that implements an object-relational data model for the core part of the MPEG-7 [6, 5] standard. It is an open and extensible system realized with the Oracle Data Cartridge technology [7]. Oracle offers with that technology a mechanism for extending the capabilities of an Oracle database (e.g., type system, query processing and indexing) for the users needs. Besides the more effective modelling of multimedia content, efficient retrieval was considered. Efficient query processing is guaranteed through new database indexing mechanisms. Innately, most database systems provide only a limited number of integrated access methods such as B-trees. Available multimedia database extension packages (e.g., DataBlades of Informix) rarely handle indexing of d-dimensional data (e.g., feature vectors [3, 8] with d>2) or advanced similarity search functionalities (e.g., k- NN search). These circumstances limit the use of database systems in multimedia. For overcoming this drawback, we introduce an Multimedia Indexing Framework (MIF) relying on the GiST framework [4]. Finally, in order to use the MMDBMS we provide query and presentation interfaces which automatically adapt to their usage environment. 2. DESCRIPTION OF THE SYSTEM The Multimedia Data Cartridge (MDC) currently consists of two parts (see figure 1). At first, the Multimedia Data Model represents the metadata describing the multimedia content relying on MPEG-7 descriptions. For this purpose, the MPEG-7 schema is mapped, with the help of Oracle s extensible type system, to a database schema, i.e., to respective object types and tables (see Appendix 1). The mapping is demonstrated on the MPEG-7 StillRegion- Type and VideoSegment which is a delegate for images and videos (e.g., StillRegion describes complete images or parts of them). Some of that attributes are declared as separate object types, or are defined as basic types and others are specific SYS.XMLType. The decision which type to use depends on the importance for the querying process. For Figure 1: Overview of the Multimedia Data Cartridge (MDC) Architecture instance, the attribute with type TextAnnotationType was chosen to be detailed further, because it is of importance for free-text search. A further important type is Scalable- ColorType which is used for storing feature vectors of color histograms extracted from images described by the StillRegionType. At second, the Multimedia Indexing Framework (MIF) provides an extensible environment for multimedia retrieval. The framework is divided into three modules (see figure 1). The GistService is realized in the external address space and runs as an own process in the Operating System environment. It manages all available access methods and offers currently support for the Generalized Search Trees (GiST [4]). The GistService may also support further access methods not relying on balanced trees (e.g., LPC-files [1]) to support efficient NN-search in high dimensional vector spaces or indexing text annotations with the help of the DependencyStructure DS [6]). The two main components are the Gist- Communicator and the GistHolder. The GistCommunicator is a COM-Object and is used for inter-process communication between the database (the GistWrapper shared library) and the implemented access methods. The GistHolder manages all currently running index trees and the accesses to them. Each index tree is identified through a global and unique ID. The GistWrapper is a shared library and is used by the database to connect to the GistService module. The GistWrapper has two tasks, namely the accessibility of the GistService through database procedures and the transfor-

2 mation of input and output data to make it usable for either the GistService or the database. The Multimedia Index Type is the necessary part in the Oracle address space. The multimedia index type consists of several indextypes, that represents all available access methods and their corresponding operators. Every indextype need an appropriate implementation (object). The object delegates all necessary index methods (e.g., ODCI- IndexInsert, ODCIIndexCreate,...) to their corresponding implementations. In MDC, most methods are forwarded to the GistService. Experimental results (see [2]) have shown effectivity and efficiency. Effectivity is guaranteed through the extension of the index services to multimedia specific operations (e.g., NN-search). Efficiency has been measured by various experiments. Our framework outperforms the Oracle Text Index for exact match queries, e.g., for data entries, 96 dimensions we achieved an average improvement factor of 14. Further information may be obtained from [2]. Figure 2: Distributed Multimedia Environment Indexing, Query and Presentation Interfaces Figure 2 shows how the MDC is embedded into the distributed system s context. Three interfaces, as marked in figure 2 are relevant for this demonstration. (1) We provide plugins for various tools which handle and produce MPEG-7 data, e.g., for Video Publisher and Movie Tool 1. The MP7 documents to be included into the MMDBMS are analyzed and then split up into objects and tables according to our data model. (2) This stored information can then be queried and presented with the help of an adaptive query and presentation interface which is connected through a web server, aware of usage environment descriptions from the clients, to the MMDBMS. It is described below. (3) Finally, we provide an interface which offers a connection to video servers where we are synchronizing the generated MP7 information (describing the query hits) with MP4 streams for enabling stream adaptation during network transfer. Let us detail the second interface. In order to serve multiple client front-ends (e.g., hand-held, PC or Palm user) the database queries are submitted through an Adaptive Query and Presentation Interface. Adaptation means the dynamic adjustement of the interface to the client s terminal capabilities (e.g., Hardware, Software). An screenshot of the Adaptive Query Interface is shown in Appendix 2. Besides the query, also the results (meta-data), and the respective streams are shown and their presentation windows are adapted. The terminal capabilities are described with the help of CC/PP which is a standardized framework developed by W3C, based on the RDF (Resource Description Format). 1 and The use of RDF enables a standard way of exchanging information about the client s capabilities. A CC/PP standard implementation, Dice, was extended and integrated into the client/server environment to adapt the query interfaces to the respective environment requirments. 3. DEMONSTRATION PLAN The intended demonstration at ACM Multimedia 2002 will consists of at least three parts. First we are demonstrating the insertion and deletion functionality of our MMDBMS. We show this for the VideoPublisher PlugIn. Second, we show the adaptive query and presentation interfaces and demonstrate their behaviour for different terminal capabilities. Different query classes are provided, ranging from simple free-text, semantically meaningful querying, (e.g., give me all goals from the team of Germany in this year s world football cup), to combined low- and high level querying (restrict the former query to matches where the German wear their traditional white/black dresses). Efficiency is the next point for demonstration. We first show the available retrieval methods (range queries, NN-search) and presents their output on a sample databases, e.g., search a given image in a 90min movie. Second, we demonstrate the data distribution and the index balance with the amdb tool (see Appendix 3), taken from the University of California, Berkeley. It will be used for creating modified and more efficient indexing methods, optimized for MPEG-7 data. 4. REFERENCES [1] G. Cha, X. Zhu, D. Petkovic, and C. Chung. An Efficient Indexing Method for Nearest Neighbor Searches in High-Dimensional Image Databases. IEEE Transaction on Multimedia, 4(1):76 87, March [2] M. Döller and H. Kosch. Enhancement of Oracle s Indexing Capabilities through GiST-implemented Access Methods. Technical report, TR/ITEC/02/2.09, University Klagenfurt, Austria, April [3] M. Flickner, H.Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker. Query by image and video content: The QBIC system. Computer, 28(9):23 32, Sep [4] Joseph M. Hellerstein, Jeffrey F. Naughton, and Avi Pfeffer. Generalized Search Trees for Database Systems. In 21st International Conference of Very Large Databases VLDB, pages , Zurich, Switzerland, [5] H. Kosch. MPEG-7 and Multimedia Database Systems. Sigmond Records, 31(2), June [6] J.M. Martinez. Overview of the MPEG-7 standard, v5.0. ISO/MPEG N4509, MPEG Requirements Group, December [7] Oracle. Data Cartridge Developer s Guide, December [8] M. A. Stricker and M. Orengo. Similarity of color images. In Storage and Retrieval for Image and Video Databases, SPIE, pages , San Jose, CA, [9] R. Tusch, H. Kosch, and L. Boeszoermenyi. VIDEX: An Integrated Generic Video Indexing Approach. In ACM Multimedia Conference 2000, pages , Los Angeles, USA, November 2000.

3 MediaFormat() Appendix 1: DB-Schema derived from MPEG-7 MDS DSType Header : SYS.XMLType id : Varchar2(4000) timebase : SYS.XMLType timeunit : SYS.XMLType mediatimebase : SYS.XMLType mediatimeunit : SYS.XMLType MovRegTempDecoType VidSegSpatTempDecoType MovRegSpatTempDecoType VidSegTempDecoType dummyattr34 : dummynttype35 dummyattr34 : dummynttype35 VSSpatioTemporalDecomposition() MRSpatioTemporalDecomposition() dummynttype35 dummyattr08 : dummynttype09 VSTemporalDecomposition() dummytype36 MovingRegion : REF MovingRegionType dummynttype09 MovingRegionRef : SYS.XMLType dummytype10 VideoSegment : REF VideoSegmentType VideoSegmentRef : SYS.XMLType StillRegSpatDecType dummyattr18 : dummynttype20 SRSpatialDecomposition() dummynttype20 dummytype21 dummynttype27 dummytype28 GridLayoutDescriptors : SYS.XMLType dummynttype16 StillRegionType dummyattr105 : dummytype85 SpatialLocator : SYS.XMLType SpatialMask : SYS.XMLType MediaTimePoint : SYS.XMLType MediaRelTimePoint : SYS.XMLType MediaRelIncrTimePoint : SYS.XMLType dummyattr23 : dummynttype27 SpatialDecomposition : dummynttype16 Image() StillRegion() dummyattr34 : dummynttype35 MRTemporalDecomposition() dummytype50 Mask : SYS.XMLType AudioDescriptor : SYS.XMLType AudioDescriptionScheme : SYS.XMLType type : SYS.XMLType SemanticBaseType dummynttype49 AbstractionLevel : SYS.XMLType Label : SYS.XMLType Definition : REF TextAnnotationType Property : SYS.XMLType MediaOccurrence : dummynttype49 Relation : dummynttype59 type_name : Varchar2(4000) type_id : Integer dummynttype59 SemanticBase() RelationType type : Varchar2(4000) source : Varchar2(4000) target : Varchar2(4000) directed : Varchar2(5) strength : Float(126) Relation() AgentType Icon : SYS.XMLType dummynttype83 TextAnnotationType VariationType Source : Varchar2(60) Source_ Content : Varchar2(60) Content_ VariationRelationship : SYS.XMLType fidelity : Real priority : Integer timeoffset : SYS.XMLType timescale : Real Variation() dummynttype03 dummynttype05 VariationDescriptionType VariationSet : dummynttype04 Variation : dummynttype05 VariationDescription() dummynttype04 VariationSetType Source : Varchar2(60) Source_ Variation : dummynttype03 VariationSet() VideoSegmentType MultimediaContentType MultimediaContent : Varchar2(60) MultimediaContent_ dummynttype07 ContentEntityType MediaInformationType dummyattr103 : dummytype85 MediaIdentification : REF MediaIdentificationType MediaProfile : dummynttype02 MultimediaContent : dummynttype07 MediaTime : SYS.XMLType MediaInformation() ContentEntity() TemporalMask : SYS.XMLType dummyattr26 : dummynttype24 dummynttype02 Mosaic : SYS.XMLType dummyattr45 : dummynttype46 MediaIdentificationType dummynttype38 Videosegment() Video() EntityIdentifier : SYS.XMLType MediaProfileType AudioDomain : SYS.XMLType VideoDomain : SYS.XMLType dummynttype24 ImageDomain : SYS.XMLType ComponentMediaProfile : dummynttype38 MediaIdentification() MediaFormat : REF MediaFormatType dummytype25 MediaTranscodingHints : SYS.XMLType MediaQuality : SYS.XMLType VisualCodingType MediaInstance : dummynttype01 Format : SYS.XMLType master : Varchar2(5) VisualTimeSeriesDescriptor : SYS.XMLType Pixel : SYS.XMLType MediaProfile() Frame : SYS.XMLType ColorSampling : SYS.XMLType MediaFormatType dummynttype01 AudioCodingType dummynttab46 Format : SYS.XMLType AudioChannels : SYS.XMLType Content : SYS.XMLType Sample : SYS.XMLType Medium : SYS.XMLType MediaInstanceType dummytype47 Emphasis : SYS.XMLType FileFormat : SYS.XMLType SpatialDecomposition : SYS.XMLType Presentation : SYS.XMLType FileSize : Integer TemporalDecomposition : REF VidSegTempDecoType System : SYS.XMLType InstanceIdentifier : SYS.XMLType SpatioTemporalDecomposition : REF VidSegSpatTempDecoType Bandwidth : Float(126) MediaSourceDecomposition : SYS.XMLType BitRate : SYS.XMLType LocationDescription : SYS.XMLType TargetChannelBitRate : Integer MediaInstance() ScalableCoding : SYS.XMLType VisualCoding : VisualCodingType AudioCoding : AudioCodingType SceneCodingFormat : SYS.XMLType GraphicsCodingFormat : SYS.XMLType OtherCodingFormat : SYS.XMLType dummyattr12 : dummynttype13 relevance : Float(126) confidence : Float(126) TextAnnotation() dummynttype13 dummytype14 FreeTextAnnotation : CLOB StructuredAnnotation : SYS.XMLType DependencyStructure : SYS.XMLType KeywordAnnotation : SYS.XMLType AnnotationTable1() MovingRegionType dummyattr106 : dummytype85 SpatioTemporalLocator : SYS.XMLType SpatioTemporalMask : SYS.XMLType dummyattr101 : dummynttype24 dummyattr97 : dummynttype98 MovingRegion() dummynttype98 dummytype99 SpatialDecomposition : SYS.XMLType TemporalDecomposition : REF MovRegTempDecoType SpatioTemporalDecomposition : REF MovRegSpatTempDecoType MediaSourceDecomposition : SYS.XMLType SemanticDescriptionType Semantics : dummynttype60 ConceptCollection : SYS.XMLType SemanticDescription() Semantics() dummynttype60 SemanticBagType dummyattr53 : dummynttype51 Graph : SYS.XMLType SemanticBag() dummynttype51 dummytype52 SemanticBase : REF SemanticBaseType SemanticBaseRef : SYS.XMLType dummytype85 dummynttype84 Semantic : REF SemanticBagType SemanticRef : SYS.XMLType dummynttype112 ColorQuantization dummyattr118 : SYS.XMLType ColorQuantization() SemanticStateType AttributeValuePair : SYS.XMLType SemanticTime : REF SemanticTimeType SemanticPlace : REF SemanticPlaceType SemanticState() EventType dummyattr56 : dummynttab57 SemanticPlace : REF SemanticPlaceType SemanticTime : REF SemanticTimeType Event() dummynttab57 dummytype58 Event : REF EventType EventRef : SYS.XMLType ColorStructureType Value(s) : CLOB colorquant : Integer ColorStructure() DominantColorType ColorSpace : SYS.XMLType ColorQuantization : SYS.XMLType SpatialCoherency : Integer Value : SYS.XMLType DominantColor() ColorSpaceType ColorTransMat : Varchar2(4000) colorreferenceflag : Varchar2(5) type : Varchar2(12) ColorSpace() SemanticTimeType Time : SYS.XMLType SemanticTimeInterval : SYS.XMLType SemanticTime() SemanticPlaceType Place : SYS.XMLType SemanticPlaceInterval : SYS.XMLType SemanticPlace() Legende: Class1 ColorLayoutType YDCCoeff : Integer CbDCCoeff : Integer CrDCCoeff : Integer dummyattr82 : SYS.XMLType ColorLayout() Mpeg7_Document Document : SYS.XMLType DB Typ Name Attribute DB Tabellen Name(n) is of type Objektreferenz Vererbung GoFGoPColorType ScalableColor : REF ScalableColorType aggregation : Varchar2(12) GoFGoPColor() ObjectType dummyattr54 : dummynttab55 Object() dummynttab55 dummytype56 Object : REF ObjectType ObjectRef : SYS.XMLType AgentObjectType Agent : AgentType AgentRef : SYS.XMLType AgentObject() Mpeg7Type timebase : SYS.XMLType timeunit : SYS.XMLType mediatimebase : SYS.XMLType mediatimeunit : SYS.XMLType Description : dummynttype06 DescriptionUnit : SYS.XMLType MetaData : Varchar2(4000) Mpeg7() dummynttype06 CompleteDescriptionType Description : Varchar2(60) Description_ID : Integer ScalableColorType Coeff : CLOB numofcoeff : Integer numofbitplanesdiscarded : Integer ScalableColor()

4 Semantic Query Interface for a client Semantic Query Interface for a stronger client Appendix 2 Input form for the Device Info

5 Appendix 3 Node View Displays bounding predicates (BPs) and items within nodes. Highlights BPs on current traversal path. Split Visualization Shows how BPs or data items are divided with PickSplit( ) Node Contents Provides textual description of node

An MPEG-7 Multimedia Data Cartridge

An MPEG-7 Multimedia Data Cartridge Mario Döller a and Harald Kosch a a Institute of Information Technology University Klagenfurt 9020 Klagenfurt, Austria Abstract Broadly used Database Management Systems