Knowledge-Driven Video Information Retrieval with LOD

Size: px

Start display at page:

Download "Knowledge-Driven Video Information Retrieval with LOD"

Rosanna George
5 years ago
Views:

1 Knowledge-Driven Video Information Retrieval with LOD Leslie F. Sikos, Ph.D., Flinders University ESAIR 15, 23 October 2015 Melbourne, VIC, Australia

2 Knowledge-Driven Video IR Outline Video Retrieval Challenges and Limitations Unstructured vs. Structured Annotations Bridging the Semantic Gap Multimedia Vocabularies and Ontologies Linked (Open) Data for multimedia Standardization of Video Annotations

3 Unstructured vs. Structured Data Plain text tags XML/XSD metadata RDFS/OWL annotations

4 Vocabularies and Ontologies Dublin Core Creative Commons Schema.org Concept definitions Classes, properties, individuals Relationships Rules

5 Video Object Representation Lightweight Annotations HTML5 Microdata JSON-LD RDFa Microformats

6 Video Object Representation Lightweight Annotations

7 Video Object Representation Lightweight Annotations

8 Video Object Representation Semi-Structured Vocabularies Vocabulary Standard Language MPEG-7 ISO/IEC XSD MPEG-21 ISO/IEC XSD NewsML IPTC NewsML-G2 XSD TV-Anytime ETSI TS XSD

9 Video Object Representation Semi-Structured Vocabularies Vocabulary Standard Language MPEG-7 ISO/IEC XSD MPEG-21 ISO/IEC XSD NewsML IPTC NewsML-G2 XSD TV-Anytime ETSI TS XSD

10 XSD to RDFS/OWL Mapping MPEG-7 COMM MPEG-7Ontos SWIntO

11 Issues Inherited from MPEG-7 Strong focus on low-level descriptors color distribution feature values of an image for red, black, and yellow still do not allow the conclusion that the image shows a sunset Boll et al., 1998

12 Issues Inherited from MPEG-7

13 Issues Inherited from MPEG-7

14 Issues Inherited from MPEG-7

15 Issues Inherited from MPEG-7

16 Issues Inherited from MPEG-7 Strong focus on low-level descriptors Conceptual ambiguity Semantic interoperability issues Syntactic interoperability issues Structural complexity: 1,182 elements, 417 attributes, and 377 complex types

17 Custom Ontologies Large Scale Concept Ontology for Multimedia (LSCOM) Linked Movie Database Ontology Multimedia Metadata Ontology (M3O) Ontology for Media Resources

18 Custom Ontologies T TBox: terminological knowledge A ABox: assertional knowledge

19 Custom Ontologies T TBox: terminological knowledge A ABox: assertional knowledge Knowledge Base

20 Custom Ontologies T TBox: terminological knowledge A ABox: assertional knowledge R RBox: role inclusion axioms + transitivity axioms

21 Multimedia Ontologies Limited DL Expressivity Ontology Language DL LinkedMDB RDFS AL LSCOM OWL AL M3O OWL SH IQ ( D) COMM OWL SH OI N ( D)

22 Multimedia Ontologies Full DL Expressivity Ontology Language DL LinkedMDB Ontology Language RDFS AL DL LSCOM VidOnt OWL OWL 2 SR AL OI Q ( D) M3O OWL SH IQ ( D) COMM OWL SH OI N ( D)

23 Domain Ontologies OWL 2 ontologies + SWRL rules Decidability

24 Domain Ontologies OWL 2 ontologies + SWRL rules Decidability Solution: OWL 2 + DL Rules + DL-safe rules: very expressive & still decidable

25 Linked Data for Multimedia Rationale Global identification with URIs Linking to annotations or media fragments in the Linked Data cloud Differentiate video objects and media fragments from information resources Support access through SPARQL queries

26 Video IR with LOD Conclusions OWL 2 ontologies are needed a) Alignment with standards Advanced b) Exploit SR OI Q ( D) constructs } inference & c) Define Rbox axioms reasoning d) Use OWL 2+DL-safe rules support LOD: video understanding, discovery,

27 Questions

28 Thank you. For more on semantic multimedia, visit

Semantic Web Test

Semantic Web Test 24.01.2017 Group 1 No. A B C D 1 X X X 2 X X 3 X X 4 X X 5 X X 6 X X X X 7 X X 8 X X 9 X X X 10 X X X 11 X 12 X X X 13 X X 14 X X 15 X X 16 X X 17 X 18 X X 19 X 20 X X 1. Which statements