Lecture 3: Multimedia Metadata Standards Prof. Shih-Fu Chang EE 6850, Fall 2002 Sept. 18, 2002 Course URL: http://www.ee.columbia.edu/~sfchang/course/vis/ EE 6850, F'02, Chang, Columbia U 1 References Digital Still Camera Image File Format Standard (Exchangeable image file format for Digital Still Cameras: Exif) - Version 2.1 http://www.exif.org/ Introduction to MPEG-7 (v2), Document: ISO/IEC JTC1/SC29/WG11 N3751. Oct. 2000. DIG35 Image Metadata Standard http://www.i3a.org/i_dig35.html S.-F. Chang, T. Sikora and A. Puri, "Overview of the MPEG-7 Standard," IEEE Transactions on Circuits and Systems for Video Technology, special issue on MPEG-7, June 2001. EE 6850, F'02, Chang, Columbia U. 2
Why Metadata Standard? Content Exchange Content owners Consumers Interoperable Client Applications Cross-operator information access Meta search engines EE 6850, F'02, Chang, Columbia U. 3 DIG 35 - image metadata EE 6850, F'02, Chang, Columbia U. 4
DIG35 Participants: Canon, Kodak, Fuji, HP, Microsoft, Polaroid, Seattle Film Works, etc. Time frame: started 1999, WD 1.0 March 00, V 1.0 Aug. 2000. Use-case Scenarios: albuming, content searching, linking, information/copyright preservation EE 6850, F'02, Chang, Columbia U. 5 DIG 35 metadata interchange model EE 6850, F'02, Chang, Columbia U. 6
DIG 35 metadata subblocks EE 6850, F'02, Chang, Columbia U. 7 EXIF: Exchangeable image file format October 1996 Version 1.0, May 1997 Version 1.1, June 1998, Version 2.1. Supported by most Digital Camera Manufacturers Consist of both image and audio file specifications Image file spec includes: Structure of image data files, Tags used by this standard, Definition and management of format versions. EE 6850, F'02, Chang, Columbia U. 8
EXIF Image File Spec Compressed files are recorded as JPEG. Uncompressed files are recorded in TIFF Rev. 6.0 A feature of Exif image files is their compatibility with standard formats in wide use today. Related attribute information for both compressed and uncompressed files is stored in the tag information format defined in TIFF Rev. 6.0. New EXIF specific attributes are stored as private tags in TIFF. EE 6850, F'02, Chang, Columbia U. 9
MPEG-7 Standard Flexible, extensible, multi-level, and standard framework for describing multimedia Systems, DDL, Video, Audio, MDS, Software Scope Feature Extraction Schedule MPEG-7 Description Search/Filtering Application Call For Proposals 10/98 Working Draft 12/99 Committee Draft 10/00 International Standard 9/01 EE 6850, F'02, Chang, Columbia U. 14
MPEG-7 Segment Types EE 6850, F'02, Chang, Columbia U. 15 MPEG-7 Framework AV Content Item signifies Data Feature to 1..* 1..* User or System 1..* 1..* describes 0..* Description Definition Language defines Description Scheme 1..* Descriptor Description Definition Language (DDL) Language to create new Ds/DSs or extend existing ones Extend XML-Schema Description Schemes (DSs) Structure and semantics of relations among Ds/DSs Descriptors (Ds) Representation of a feature of AV data EE 6850, F'02, Chang, Columbia U. 16
MPEG-7 Application Chain MM Content Description Generation Description Definition Language (DDL) User or data processing system Description Schemes (DS) MPEG7 Description Search / Descriptors (D) Query Engine Filter Agents Encoder MPEG7 Coded Description Decoder EE 6850, F'02, Chang, Columbia U. 17 Parts of MPEG-7 (ISO/IEC 15938) Systems Binary encoding, Dynamic update, Transport, Synchronization, and IPMP tools Description Definition Language (DDL) Language for defining new, extending existing DSs and Ds Visual Visual Ds and DSs Audio Audio Ds and DSs Multimedia Description Schemes (MDS) Generic Ds and DSs; neither purely visual nor purely audio Reference Software Conformance EE 6850, F'02, Chang, Columbia U. 18
X M L extensible Markup Language (XML) Derived from SGML (Standard Generalized Markup Language) Description of structure and semantics of documents Human- and machine- readable Author-defined elements and attributes: DTD or XML-Schema <customer id="ana2000"> <name> Ana Benitez </name> <address country="us"> <street>500 W 120</street> <city> New York </city> <state> New York </state> <postal> 94571 </postal> </address> </customer> EE 6850, F'02, Chang, Columbia U. 19 XML / DTD / XML-Schema XML Description DTD Definition <customer id="ana2000"> <name> Ana Benitez </name> <address country="us"> <street>500 W 120</street> <city> New York </city> <state> New York </state> <postal> 94571 </postal> </address> </customer> <!ELEMENT customer (name, email?, address+) > <!ATTLIST customer id ID #REQUIRED> <!ELEMENT name (#PCDATA)> <!ELEMENT email (#PCDATA)> <!ELEMENT address (street, city, state, postal)> <!ATTLIST address country CDATA #REQUIRED> <!ELEMENT street (#PCDATA)> <!ELEMENT city (#PCDATA)> <!ELEMENT state (#PCDATA)> <!ELEMENT postal (#PCDATA)> EE 6850, F'02, Chang, Columbia U. 20
XML / DTD / XML-Schema (cont) XML Description <customer id="ana2000"> <name> Ana Benitez </name> <address country="us"> <street>500 W 120</street> <city> New York </city> <state> New York </state> <postal> 94571 </postal> </address> </customer> XML-Schema Definition <complextype name= customer > <element name= name type= string /> <element name= email type= string minoccurs= 0 /> <element name= address type= addresstype maxoccurs= unbounded /> <attribute name= id type= ID use= required /> </complextype> <complextype name= address > <element name= street type= string /> <element name= city type= string /> <element name= state type= string /> <element name= postal type= positiveinteger /> <attribute name= country type= string /> </complextype> Some Useful Sites W3C: http://www.w3.org/xml XML Cover Pages: http://www.oasis-open.org/cover/ Web Developer s Virtual Library: http://wdvl.com/ XML Industry Portal: http://www.xml.org/ XML Schemas Endgame: http://www.xml.com/pub Apache XML Project: http://xml.apache.org/ IBM alphaworks: http://www.alphaworks.ibm.com/ EE 6850, F'02, Chang, Columbia U. 22
Video Descriptors Color Dominant Color Scalable Color Color Layout Color Structure GoF/GoP Color Texture Homogeneous Texture Texture Browsing Edge Histogram Shape Region Shape Contour Shape 3D Shape Motion Camera Motion Motion Trajectory Parametric Motion Motion Activity Localization Region Locator Spatio-Temporal Locator Other Face Recognition EE 6850, F'02, Chang, Columbia U. 23 Example: Color Histogram <GoFGoPHistogram HistogramTypeInfo = "Average"> <ColorHistogram> <ColorSpace> <HSV/> </ColorSpace> <ColorQuantization ColorQuantizationType = uniform"> <bin_number> 4 </bin_number> <bin_number> 4 </bin_number> <bin_number> 4 </bin_number> </ColorQuantization> <Histogram HistogramNormFactor = "1" NumberHistogramBins = "64"> <HistogramValue> 444 </HistogramValue> <HistogramValue> 34 </HistogramValue> <HistogramValue> 58 </HistogramValue> <HistogramValue> 564 </HistogramValue> <HistogramValue> 16 </HistogramValue> <! -- Other HistogramValue elements -- > </Histogram> </ColorHistogram> </GoFGoPColorHistogram> EE 6850, F'02, Chang, Columbia U. 24
Structure Description Tools TextAnnotation D SpatialMask D... StillRegion DS Segment Relation DS VideoSegment DS MovingRegion DS... Segment DS describes Multimedia Content Segment Decomposition DS EE 6850, F'02, Chang, Columbia U. 25 Structure Description (I) Video Segment Segment Decomposition Video Segments MediaTime Mosaic GoFGoPColor TextAnnotation Segment Decomposition Moving Region Segment Decomposition Moving Regions MediaTime ScalableColor ParametricMotion TextureBrowsing ContourShape TextAnnotation Relation above
Structure Description (II) Still region SR1: Creation inform a tion Text annotation <StillRegion id="sr1"> <TextAnnotation> <FreeTextAnnotation> Alex shakes hands with Ana </FreeTextAnnotation> </TextAnnotation> <SpatialDecomposition overlap="false" gap="true"> Still region SR3: Text annotation Matching hint Color structure Spatial segment decompos i tion: No overlap, gap Still region SR2: Text annotation Color structure Directional spatial segment relation: left <StillRegion id="sr2"> <TextAnnotation> <FreeTextAnnotation> Alex </FreeTextAnnotation> </TextAnnotation> <VisualDescriptor xsi:type="colorstructuretype">... </VisualDescriptor> </StillRegion> <StillRegion id="sr3"> <TextAnnotation> <FreeTextAnnotation> Ana </FreeTextAnnotation> </TextAnnotation> <MatchingHint> <Hint value="0.455" xpath=../../visualdescriptor"/> </MatchingHint> <Relation xsi:type="directionalspatialsegmentrelationtype name="left target="#sr2"/> <VisualDescriptor xsi:type="colorstructuretype">... </VisualDescriptor> </StillRegion> </SpatialDecomposition> </StillRegion> Semantic Description Tools... AbstractionLevel AgentObject DS Object DS AnalyticModel DS Segment DS Semantic Relation DS SemanticBase DS Event DS Concept DS Narrative World SemanticBag DS SemanticState DS SemanticPlace DS Multimedia Content captures describes Semantic DS SemanticTime DS EE 6850, F'02, Chang, Columbia U. 28
Semantic Description Segment-semantic base relation: hasmediaperceptionof Segment-semantic base relation: hasmediasymbolof Concept C1: Property Property Semantic place-semantic base relation: haslocationof Concept-semantic base rel a tion: haspropertyof SemanticPlace SP1: Place SemanticTime ST1: Time New York 9 September Semantic time-semantic base relation: hastimeof Comradeship Shake hands Alex Ana Event EV1: Object-event relation: hasaccompanierof Segment-semantic base relation: hasmediaperceptionof Object-event relation: hasagentof Agent object AO1: Person Agent object AO2: Person EE 6850, F'02, Chang, Columbia U. 29 An MPEG- 7 Descri ption Creation information: Creation Creator Creation corrdinates Creation location Creation date Photographer: Seungyup Place: Columbia University Time: 19 September 1998 Still region SR1: Creation inform a tion Text annotation Spatial segment decompos i tion: No overlap, gap Still region SR2: Text annotation Color structure Media information: Media profile Media format Media instance 704x480 pixels True color RGB http://www.ee.columbia.edu/~ana/alex&ana.jpg Segment-semantic base relation: hasmediasymbolof Object-event relation: hasagentof Comradeship Shake hands Alex Usage unformation: Rights Ana Columbia University, All rights reserved Concept C1: Property Property Concept-semantic base rel a tion: haspropertyof Object-event relation: hasaccompanierof Event EV1: Semantic time Semantic place Still region SR3: Text annotation Matching hint Color structure Directional spatial segment relation: left Segment-semantic base relation: hasmediaperceptionof Agent object AO1: Person Agent object AO2: Person
Multimedia Description Schemes Content organization Collections Models User interaction Media Structure Creation & Production Content management Content description Semantics Usage Navigation & Access Summaries Views Variations User Preferences User History Basic elements Schema Toolss Basic Datatypes Links & Media Localization Basic Tools EE 6850, F'02, Chang, Columbia U. 31 Other MDS Creation & Production Description of content creation and production (e.g. title and creator), mostly author-generated Usage Description of usage of the content (e.g. rights holders and publication) Media Description of instances of storage media (e.g. storage format) for AV content EE 6850, F'02, Chang, Columbia U. 32
Other MDS Categories Navigation & Access Description of summaries (hierarchical and sequential) and views for efficient browsing Description of variations for personalized access Translation, transcription, reduction, etc. Content Organization Description of collections, classifications, and models User Interaction Description of user s preferences pertaining to consumption of multimedia material EE 6850, F'02, Chang, Columbia U. 33 Application APIs Schema Decoder BiM/Textual Decoding Description Decoder Reconstruction BiM/Textual Parsing Compression Layer MPEG-7 Terminal Architecture (March 01) Schema streams Defines Description streams Multimedia streams Describe Upstream Data Elementary Streams Demultiplex Demultiplex Demultiplex MPEG-2 IP ATM MP4... Multiplexed Streams Delivery Layer Transmission/Storage Medium
Related Course Projects Survey of GPS attributes and their potential use in image organization Visualization tools for MetaData streams EE 6850, F'02, Chang, Columbia U. 35