Event-Based Modeling and Processing of Digital Media

Transcription

1 Event-Based Modeling and Processing of Digital Media Rahul Singh Zhao Li Pilho Kim Derik Pack Ramesh Jain Experiential Systems Group Georgia Institute of Technology

2 Ubiquity of Media Surveillance, biometrics, situation monitoring, Digital Information Processing Research Genomics, Proteomics, Structural Biology Database Research Biomedical imaging, medical records Personal Media

3 Motivation Media Dependent Features and Similarity Functions Digital Information Processing Research Images Molecular Index Database Research Text Audio

4 Problem Formulation Text Images Semantic correlations between media are important! A General Model To Semantically Unify Multimedia

5 Goals Provide a model to unify multimedia semantically Storage scheme for efficient queries of structured and unstructured data Semi-Automatic signal processing and tagging of media

6 Outline I. Event Model II. Event Storage III. Event Tagging System IV. Encompassing Architectures V. Conclusions

7 Event Model

8 Event-Based Unified Modeling of Multimedia Media 1 Images Physical Event Media 2 Video Media 3 Event Model Text Media n Audio Definition: An event is an observed physical reality parameterized by space and time. The observations describing the event are defined by the nature or physics of the observable, the observation model, and the observer.

9 The Conceptual Model Event Event Information } Time Location Participants Event Relations } Spatio-Temporal Relations Event Aggregations Media Support } Media Type(s) Media Locators, Indexes

10 Modeling Time Discrete time-stamping does not work very well Direct application of interval based models is not possible for many types of media My Trip to Paris for CVDB June 10 June 11 June 12 June 14 Assign to events a mixed (point or interval) character Point-Point Relations, Point-Interval Relations, Interval-Interval Relations Before Simultaneous After Time Before During After Time Before Equal Meet Overlap During Start Finish Time

11 Motivating Example--Meetings Why? Significant research into smart meeting rooms Need to facilitate remote participation Various media can be found in meetings

12 Event Storage

13 Event Storage in Meetings A Closer Look at Meeting Data Static Information such as the name of the attendee etc. Dynamic Information such as the process of the meeting. Most of static information is structured and most dynamic information is unstructured. Our Approach Using relational database approach to handle the static information. Using the XML database approach to deal with the dynamic information.

14 Meeting System ER&E Diagram -- Entities Relationship and Event Diagram Time Title Event infrastructure Compound, Sub-events Location M Static Info. Meeting Dynamic Info. Event Relationship (Temporal, causality) Name Attendee N Attend Related Multi-silo Data resources (video, audio, PowerPoint) Birthday Position Structured Data Unstructured Data

15 Tables of The Meeting Information System Meeting MT# Time Location MeetingProcessing (XMLType) Attendee E_mail FName LName Position Image Attend Relationship E_mail MT#

16 XML Schema for Meetings

17 Queries Relational/XML Query select extract((e.meetingprocessing), '/meeting/presentation/how/ask_question/video') from meeting e where meetinglocation='tsrb1 ; XML Query select extract(value(e), '//MEETING[Location= TSRB1 ]/Presentation/how/ask_question /video') from xmldemotable e";

18 Performance Evaluation

19 Event Tagging System

20 Example of Meeting Related Event Flow Setup Meeting Meeting Capture Post Meeting Discussion Time Line Identify Member Presentation Material Different Color means Different types of data

21 How to combine manual annotation to automatic feature detection in network Data Event Detector Symbol Candidate Domain Analysis Data Network Assign Teaching Value Manual Event Tagging Symbol On-Line Processing Symbol Network Off-Line Processing Both

22 Speaker Identification Flow Specific Microphone in Specific Environment Limited Speakers Background Noise Filtering Speech Detection Extract Vocal Feature Compare with Speaker Database Prior Assumptions of Background Noise Prior Assumptions of Speech Signal Features Prior Assumptions of Acoustic Features Prior Knowledge on Participants Speaker Candidate

23 Encompassing Architectures

24 Architecture Diagram Continuous Query Server Sensors Event Detector Client Domain EventBase Data Silo Update Server

25 Conclusions

26 Conclusions Event model handles complex spatio-temporal reasoning and the semantic unification of heterogeneous multimedia Event tagging incorporates humans in the event detection process Combining structured and unstructured meta-data improves efficiency in event storage

27 Future Work Non ad-hoc approaches for processing techniques over multiple media types Effect of top-down event approach on signal processing and computer vision Researching system implementations of unified multimedia models

28 Questions?