EEG analysis for implicit tagging of video data

Size: px

Start display at page:

Download "EEG analysis for implicit tagging of video data"

Toby Shelton
5 years ago
Views:

1 EEG analysis for implicit tagging of video data Sander Koelstra, Christian Mühl, Ioannis Patras MultiMedia and Vision Group, Queen Mary, University of London, UK Human Media Interaction, University of Twente, NL September 9, 2009 ABCI Workshop / 14

2 New multimedia data is generated at amazing rate YouTube: 20 hours uploaded per minute Flickr: about 4000 new photo s per minute Organization and annotation of data is still mainly manual Efficient indexing and retrieval of multimedia is a huge challenge ABCI Workshop / 14

3 Approaches to solving this problem Automated tagging is one possible solution Works quite well for simple concepts (i.e. car, face) Higher level concepts often still elude the grasp of automated content analysis Involve the user Active involvement (i.e. tagging through games) Implicit tagging (observe user while watching videos) EEG is one possible modality for passive, implicit tagging ABCI Workshop / 14

4 EEG-based implicit tag validation Here we propose implicit tag validation using EEG signals User is shown a video and a tag, goal is to predict whether the video matches the tag Possible applications: Validate user-contributed tags Use as feedback mechanism for automated tagging or retrieval ABCI Workshop / 14

5 ABCI Workshop / 14

6 N400 mismatch negativity The component exhibits a higher negativity when semantic mismatches occur in stimulus example: He spread the bread with socks Hypothesis: The N400 occuring after the display of a tag will show a higher negativity when the tag does not match the video ABCI Workshop / 14

17 subjects were recorded, of which 12 were male. Ages ranged from 19 to 31, with a mean age of 25.

7 17 subjects were recorded, of which 12 were male. Ages ranged from 19 to 31, with a mean age of videos, 7 categories with 7 videos per category Each video was shown twice in randomized order, once followed by a matching tag and once by a non-matching tag ABCI Workshop / 14

8 Category/Label Airplane take off People kissing People getting out of cars Mice drinking water Cats opening doors Jawdrop Laughing people Source Plane spotter homevideos Hollywood films Hollywood films Mouse behaviour dataset Pet homevideos facial expression database meeting dataset Earlier research indicates that the categories of faces, animals and inanimate objects can be separated reasonably well by EEG analysis of watching subjects Most categories depict natural videos, rather than labaratory settings ABCI Workshop / 14

9 Independent Component Analysis used to remove artefacts (blinks, movements, power line interference etc.). Blink component ABCI Workshop / 14

10 Independent Component Analysis used to remove artefacts (blinks, movements, power line interference etc.). ERP component ABCI Workshop / 14

Experiment results (1) Repeated measures ANOVA used to determine significance of signal difference 10 of 32 electrode locations show a significantly more negative signal during this period Electr.

11 Experiment results (1) Repeated measures ANOVA used to determine significance of signal difference 10 of 32 electrode locations show a significantly more negative signal during this period Electr. F(1, 16) p-value µv CP Pz CP Cz PO C F T FC AF ABCI Workshop / 14

12 Experiment results (2) ABCI Workshop / 14

13 We are currently performing an analysis on single trials in order to be able predict the validity of novel tags Daunting task due to large variability of signals Highest success rate in a subject achieved so far is 75.5% Using a combination of Common Spatial Patterns and a Sparse Multinomial Logistic Regression Classifier Tested using leave-one-out cross-validation on a single subject ABCI Workshop / 14

14 Thank you for listening... ABCI Workshop / 14

15 ABCI Workshop / 14

Sentiment analysis under temporal shift

Sentiment analysis under temporal shift Jan Lukes and Anders Søgaard Dpt. of Computer Science University of Copenhagen Copenhagen, Denmark smx262@alumni.ku.dk Abstract Sentiment analysis models often rely