Computer-Aided Semantic Annotation of Multimedia. Project Presentation. June 2008

Transcription

1 Computer-Aided Semantic Annotation of Multimedia Project Presentation June 2008

2 Presentation Outline 1. Introduction 7. Partner Roles 2. Consortium 8. Research Areas 3. Motivation 9. Project Expected Results 4. Project Main Target 10. Potential Impact 5. Project Objectives 11. Contact Information 6. Approach June 2008 CASAM Project Presentation 2

3 Introduction Project Acronym / Number: CASAM / FP Project Title: Computer-Aided Semantic Annotation of Multimedia Project Website: Project Logo: Total Cost / EC Funding: 4.24 m / 3.03 m Duration: 1 April March 2011 June 2008 CASAM Project Presentation 3

4 Consortium 1 INTRASOFT International SA (INTRASOFT) Project Coordinator 2 National Centre of Scientific Research Demokritos (NCSR) 3 University of Birmingham (UNI-BHAM) 4 Technische Universität Hamburg (TUHH) 5 Athens Technology Center SA (ATC) 6 Deutsche Welle (DW) 7 Agência de Notícias de Portugal (LUSA) 8 European Journalism Centre (EJC) June 2008 CASAM Project Presentation 4

5 Motivation Multimedia Content Annotation Machine Annotation (fully automated) CASAM Manual Annotation Annotation of multimedia by employing human-machine synergy that optimizes user effort. Pressing need for annotated multimedia content Immature machine-only (fully automated) annotation Cost-prohibitive manual (human) annotation Computer-Aided Semantic Annotation of Multimedia User effort optimization through human-machine synergy Human and machine intelligence combination June 2008 CASAM Project Presentation 5

6 Project Main Target To design, implement and validate an Annotation Toolkit that will augment machine knowledge with human input, towards the target of minimizing user effort and bridging the gap between machine-derived and human annotation 3-phases Platform Validation in terms of easiness and quality of annotation: Initial System Architecture First Prototype Second Final Prototype June 2008 CASAM Project Presentation 6

7 Project Objectives To provide the technology for Multimedia Analysis Knowledge-driven analysis Reassessment of context and concepts based on user feedback and new knowledge To develop novel methods of Reasoning for Multimedia Interpretation Built around information exchange between multimedia analysis and human To design unique intelligent Human-Computer Interaction methods Maximization of the expected information gain from the user s input Cooperative environment to guide the knowledge aggregation process June 2008 CASAM Project Presentation 7

8 Approach (I) Introduction of the effort-optimized knowledge aggregation concept Annotation toolkit featuring a human-machine interaction loop for fast convergence of human and machine knowledge Annotation task into 3 main operations: Reasoning for Multimedia Interpretation (RMI) Knowledge-Driven Multimedia Analysis (KDMA) Human-Computer Interaction (HCI) Demonstration Domain: News Production in News Agencies and Broadcasters Methods also applicable to other domains June 2008 CASAM Project Presentation 8

9 Approach (II) June 2008 CASAM Project Presentation 9

10 Approach (III) CASAM information flow (interaction loop) 1. KDMA analyzes the multimedia content (video, image, natural language) and extracts the low level information. 2. User enters a small number of keywords that describe the high-level concepts present in the multimedia content. 3. RMI augments the KDMA-derived and user-provided information instantiating an ontology, infers new concept instances and reassesses the context and previous input from KDMA. Results are fed back to the KDMA for multimedia analysis driven by the renewed information until nothing more can be inferred by RMI or recognized by KDMA. 4. RMI reasons about what is needed in order to add missing instances or resolve any ambiguities. If the annotation target has been achieved the loop exits, otherwise the information requirements are fed to HCI. 5. HCI transforms the information requirements to input requests towards the user, optimizing the user interaction. The user interface also provides the user with the opportunity to alter the knowledge acquisition path devised by the system. 6. HCI input is passed to the RMI and the loop continues from step 3. June 2008 CASAM Project Presentation 10

11 Partner Roles Project Management (INTRASOFT) Platform Integration ATC EJC INTRASOFT KDMA Platform Evaluation NCSR RMI TUHH HCI User Requirements & Knowledge Representation UNI-BHAM End Users DW, LUSA Dissemination & Exploitation (ATC) June 2008 CASAM Project Presentation 11

12 Research Areas (I) Knowledge-Driven Multimedia Analysis Text Processing Image Processing Video Processing and Video-OCR Audio Processing Ontology-based Information Extraction June 2008 CASAM Project Presentation 12

13 Research Areas (II) Knowledge Representation and Reasoning for Multimedia Interpretation Probabilistic Reasoning Machine Learning Multimedia Interpretation Generation of Requests for Information June 2008 CASAM Project Presentation 13

14 Research Areas (III) Interactive Interface Design, Personalization and Optimization Effective User Interfaces for Multimedia Tagging Adaptive Interfaces Domain Models User Models June 2008 CASAM Project Presentation 14

15 Project Expected Results Unified representation of related domain knowledge Methodology for knowledge-driven multimedia analysis Methodology for reasoning for multimedia interpretation Methodology for intelligent human-computer interface The final toolkit for computer-aided semantic annotation of multimedia content June 2008 CASAM Project Presentation 15

16 Commercial Benefits Potential Impact (I) Creators to be able to design more participative and communicative forms of content Publishers in creative industries, enterprises and professional sectors to increase their productivity with innovative content of greater complexity and ease of repurposing Organizations to be able to automate the collection and distribution of digital content and machine traceable knowledge and share them with partner organizations June 2008 CASAM Project Presentation 16

17 Potential Impact (II) Scientific Impact Deployment of: Self-adaptive, self-aware, non-isolated and knowledgeguided Multimedia Analysis methods Intelligent Reasoning methods, guided by multiple targets Human-Computer Interaction methods, maintaining the user model and customize the queries June 2008 CASAM Project Presentation 17

18 Contact Information Coordinator contact details: Dr. Antonis Ramfos INTRASOFT International S.A. Tel: Fax: Dissemination responsible: Dr. George Kountourakis Athens Technology Center S.A. Tel: Fax: Project Website: June 2008 CASAM Project Presentation 18

19 June 2008 CASAM Project Presentation 19