TIB AV-Portal Challenges managing audiovisual metadata encoded in RDF

Transcription

1 Challenges managing audiovisual metadata encoded in RDF Jörg Waitelonis yovisto GmbH Margret Plank German National Library of Science and Technology (TIB) Hannover Prof. Dr. Harald Sack HPI-Potsdam / FIZ Karlsruhe & KIT SWIB16 Semantic Web in Libraries Conference 2016, November 2016, Bonn, Germany

2 WELCOME SWIB16 Semantic Web in Libraries yovisto SWIB16 Christian Hentschel Prof. Dr. Harald Sack Jörg Waitelonis What we do: Based in: Intelligent Linked Data-, Ontology- and Metadata-Management August Bebel Str Potsdam Germany Knowledge Discovery & Knowledge Mining Video- & Image Analysis, User Interfaces, Visualization 2

3 Developed in cooperation of: German National Library of Science and Technology (TIB), Hannover Hasso-Plattner-Institute for IT-Systems Engineering (HPI), Potsdam Hosted and maintained by: yovisto GmbH, Potsdam flowworks GmbH, München (Asset Management, Playout) 3

4 > 8000 Lectures, Conference talks, Interviews, Simulations, Visualizations, Research Data for Scientist, Lecturers, Teachers, Learners 4

5 Users, Customers, Uploader Video Upload TIB Curators Manage Metadata: DOI, QA, Right clearance Search View Ingest Media Asset Management Streaming ve ap d o pr Workflow Management AV-Analysis: Video Segmentation Optical Character Recognition (OCR) Speech-to-text (ASR) Visual Concept Detection (VCD) Search Index RDF Triplestore Semantic Analysis: Context Modelling Named Entity Linking 5

6 Semantic metadata analysis with Named Entity Linking Textual Metadata Authoritative: Non-authoritative: Formal, descriptive, technical E.g. title, description, keywords, etc. Manually authored Refers to entire video (coarse grained) E.g. ASR/OCR-transcripts Automatically Extracted Refers to fragments of the video (fine grained) mapping mapping Knowledge base GND subject headings GND = Gemeinsame Normdatei (Integrated authority file) Incl. English translations from mappings to DBpedia, LCSH, MACS and WTI Thesaurus [1] Sven Strobel, PalomaMarín-Arraiza: Metadata for Scientific Audiovisual Media: Current Practices and Perspectives of the, In Proc. of Metadata and Semantics Research: 9th Research Conference, MTSR 2015, Manchester, UK, 2015, Springer 6

7 7

8 The Data Model & RDF-Export Why RDF? How? Extensible Different serialization forms Interoperable Queryable (SPARQL) W3C standard Vocabulary selection Problem: heterogeneous metadata authoritative, spatio-temporal, nested annotations Vocab discovery -> Selection criterions [2] Jörg Waitelonis, Margret Plank, Harald Sack, : Integrating Automatically Generated Video Annotations into the Web of Data, in Proc. of 20th International Conference on Theory and Practice of Digital Libraries (TPDL 2016) 8

9 The Data Model & RDF-Export Vocabulary Selection Issues: Availability on the Web Openness Level of complexity/richness Maintained Trustworthy authorship Usage by others / popularity Documentation Adequate meaning Specificity Datatypes Avoid contradictions, e.g. Domain & range / sub- & super-class Datatype vs. object properties Does it fit currently used models cf. 9

10 The Data Model & RDF-Export Standard Metadata and Basic Structure DCMI Metadata Terms DCMI Type Vocabulary schema.org Vocabulary Friend of a Friend Vocabulary 0.1 Bibframe Vocabulary tib:video/16453 schema:name schema:description schema:keywords schema:datecreated schema:duration rdf:type schema:url schema:producer schema:publisher schema:license schema:availability bibframe:doi schema:thumbnailurl "Wall-crossing and geometry at infinity of Betti moduli spaces"@en ; "Linear algebraic differential equation (in one variable)..."@en ; "Betti moduli"@en, "chaos theory"@en, "singularity"@en ; " T00:00:00+01:00"^^< ; 1:16:48 ; schema:movie ; < ; gnd: ; tib:institut_des_hautes tudes_scientifiques_%28ih_s%29 ; < ; schema:onlineonly ; < ; < 10

11 The Data Model & RDF-Export Spatio-temporal Metadata Open Annotation Data Model (OA) NLP Interchange Format (NIF) tib:video/16453#t=smpte-25:0:05:00:22,0:05:03:00dcterms:ispartof tib:video/ tib:asr/16453_ _7522 oa:hastarget tib:video/16453#t=smpte-25:0:05:00:22,0:05:03:00 ; oa:annotatedby tib:annotator/asr ; rdf:type oa:annotation ; oa:hasbody tib:asr/16453_ _7522#char=0,5617. tib:asr/16453_ _7522#char=0,5617rdf:type nif:context ; rdf:type nif:rfc5147string ; nif:isstring "... five sets...". tib:asr/16453_ _7522#char=4743,4747nif:referencecontext tib:asr/16453_ _7522#char=0,5617 ; itsrdf:taidentref gnd: ; itsrdf:taannotatorsref tib:annotator/nel ; rdf:type nif:phrase ; rdf:type nif:string ; nif:beginindex "4743" ; nif:beginindex "4747" ; nif:anchorof "sets". 11

12 The Data Model & RDF-Export Open Annotation tib:video/16453#t=smpte-25:0:23:12:12,0:23:14:4 oa:hastarget tib:annotator/asr oa:annotatedby tib:asr/16453_ rdf:type ao:annotation oa:hasbody... the astronaut gnd: x 12

13 The Data Model & RDF-Export NLP Interchange Format (NIF) tib:video/16453#t=smpte-25:0:23:12:12,0:23:14:4 oa:hastarget tib:annotator/asr oa:annotatedby tib:asr/16453_ rdf:type ao:annotation oa:hasbody nif:context nif:rfc5147string rdf:type tib:asr/16453_ #char=0,62 nif:isstring... the astronaut nif:referencecontext nif:string nif:phrase rdf:type tib:asr/16453_ #char=23,32 itsrdf: taldentref nif: nif: beginindex endindex nif:anchorof astronaut itsrdf: taannotatorsref gnd: x tib:annotator/nel

14 Data Quality Standard metadata Automatically created metadata Concise manual verification and clearing by TIB subject specialists use the verified information to improve subsequent analysis AV analysis typical detection errors (ASR, OCR, etc.) Semantic analysis missing annotations wrong annotations knowledgebase errors and insufficiencies 14

15 Data Quality: Video Text Recognition Title: "Lecture on Science and Creativity" Author: Kroto, Harold Improving OCR Extend OCR vocabulary (per video) with subject specific terminology terminology from manually verified metadata OCR detects Kyoto Before OCR: extend the OCR language model & subsequent spell-correction with terms from authoritative metadata (e.g.: Creativity, Harold, Kroto, Lecture, Science) OCR now detects Kroto. 15

16 Data Quality: Video Text Recognition Title: Armstrong landed on the moon NEL: dbp:neil_armstrong, dbp:moon Improving OCR Extend OCR vocabulary (per video) with subject specific terminology terminology from manually verified metadata related terms from manually verified metadata (with help of NEL & graph-traversal/fact ranking) Apollo11 Saturn V Rocket OCR detects Pocket Before OCR: extension/boosting with related technical terms, e.g.: dbp:apollo_11, dbp:buzz_aldrin, dbp:nasa, dbp:rocket, dbp:saturn_v-b, dbp:low_earth_orbit, etc. OCR now detects Rocket. apolloarchive.com 16

17 Data Quality Semantic Analysis Multiple entries for the same entity Knowledgebase errors and insufficiencies: Sparse linking Insufficient for graph-based disambiguation Multiple entries for the same entity e.g. Harald Sack Harald Sack Solution? Manual intervention (e.g. blacklisting)? 17

18 Data Quality: Semantic Analysis Typical Annotation Errors 01 Missing: 02 Compound Split Raman carried out ground-breaking work in the field of light scattering, which earned him the 1930 Nobel Prize in Physics. Terms that have not been annotated dbp:nobel_price 02 Compound Split: Entities split into two separate entities 03 General/Specific: A more general instead of the more specific entity has been chosen 04 Wrong Entity Wrongly annotated entities not classified in category 1-3 The best NEL tools reach F1-measures barely beyond 0.6 & dbp:physics vs. dbp:nobel_price_in_pysiscs 03 General/Specific After serving the US Army, he was again readmitted and even studied under John von Neumann. dbp:army vs. dbp:united_states_army (cf. [3]) 04 Wrong Entity Alternative? Manual annotation? In 1909, Polanyi studied to be a physician, obtaining his medical diploma in dbp:michael_polanyi_center vs. dbp:michael_polanyi 18

19 Data Quality: Semantic Analysis Asked 20 people and a machine to annotate the same text: human machine 01 Missing: 02 Compound Split: 03 General/Specific: 04 Wrong Entity It seems that machine automated NEL can significantly support the user in localizing entities, human and users can support the machines in the actual disambiguation process Solution? Semi-automatic annotations! [4] Tabea Tietz, Joscha Jäger, Jörg Waitelonis, Harald Sack: Semantic Annotation and Information Visualization for Blogposts with refer In Proc. of the Second International Workshop on Visualization and Interaction for Ontologies and Linked Data (VOILA '16), volume 1704 pages 28-40,

20 Data Quality: Semantic Analysis Semi-automatic annotation and semantic enrichment tools, e.g.: 20

21 Data Quality: Semantic Analysis Semi-automatic Annotation: Improve NEL algorithm by learning from manually corrected annotations Extend the NEL systems knowledge base with information derived from the manually corrected annotations add links new to the KB graph between entities of the same document add new surface-forms to the candidate mapping dictionary AIDA 16.2 % IITB 14.0 % KORE % MSNBC 6.4 % NEEL % OKE % REUTERS 16.5 % RSS % SPOTLIGHT 12.6 % Improvements on repeating annotations (f-measure, standard NEL benchmark datasets [3]) 21

22 Summary exports all metadata as RDF Many quality issues still present in automatically extracted metadata NEL can also improve audio-visual extraction methods semi-automatic annotation tools can compensate machine and human mistakes NEL can be improved by learning from manually corrected annotations Better structure and open up the content! [6] Janna Neumann, Margret Plank: TIB's Portal for audiovisual media: New ways of indexing and retrieval In IFLA Journal March : 17-23, doi: /

23 References [1] Sven Strobel, PalomaMarín-Arraiza: Metadata for Scientific Audiovisual Media: Current Practices and Perspectives of the, In Proc. of Metadata and Semantics Research: 9th Research Conference, MTSR 2015, Manchester, UK, 2015, Springer [2] Jörg Waitelonis, Margret Plank, Harald Sack: : Integrating Automatically Generated Video Annotations into the Web of Data, in Proc. of 20th International Conference on Theory and Practice of Digital Libraries (TPDL 2016) [3] Ricardo Usbeck, Michael Röder, Axel-Cyrille Ngonga Ngomo, Ciro Baron, Andreas Both, Martin Brümmer, Diego Ceccarelli, Marco Cornolti, Didier Cherix, Bernd Eickmann, Paolo Ferragina, Christiane Lemke, Andrea Moro, Roberto Navigli, Francesco Piccinno, Giuseppe Rizzo, Harald Sack, René Speck, Raphaël Troncy, Jörg Waitelonis und Lars Wesemann: GERBIL -- General Entity Annotation Benchmark Framework in 24th WWW conference [4] Tabea Tietz, Joscha Jäger, Jörg Waitelonis, Harald Sack: Semantic Annotation and Information Visualization for Blogposts with refer In Proc. of the Second International Workshop on Visualization and Interaction for Ontologies and Linked Data (VOILA '16), volume 1704 pages 28-40, 2016 [5] Janna Neumann, Margret Plank: TIB's Portal for audiovisual media: New ways of indexing and retrieval In IFLA Journal March : 17-23, doi: /

24 Thank You! Challenges managing audiovisual metadata encoded in RDF Jörg Waitelonis SWIB16 Semantic Web in Libraries Conference 2016, November 2016, Bonn, Germany Conception & Prototyping Software Development UI/UX Design Consulting & Operations